U.S. Department of Education 
Miguel Cardona 
Secretary 


Institute of Education Sciences 
Mark Schneider 
Director 


National Center for Education Evaluation and Regional Assistance 
Matthew Soldner 
Commissioner 


Marsha Silverberg 
Project Officer 


November 2021 


The Institute of Education Sciences (IES) is the independent, non-partisan statistics, research, and 
evaluation arm of the U.S. Department of Education. The IES mission is to provide scientific evidence 
on which to ground education practice and policy and to share this information in formats that are 
useful and accessible to educators, parents, policymakers, researchers, and the public. 


We strive to make our products available in a variety of formats and in language that is appropriate to a 
variety of audiences. You, as our customer, are the best judge of our success in communicating 
information effectively. If you have any comments or suggestions about this or any other IES product 
or report, we would like to hear from you. Please direct your comments to ncee.feedback@ed.gov. 


This report was prepared for the Institute of Education Sciences (IES) under Contract ED-IES-12-C-0087 
by Abt Associates and Mathematica. The content of the publication does not necessarily reflect the 
views or policies of IES or the U.S. Department of Education nor does mention of trade names, 
commercial products, or organizations imply endorsement by the U.S. Government. 


This report is in the public domain. While permission to reprint this publication is not necessary, it 
should be cited as: 


Linkow, T., Parsad, A., Martinez, A, & Miller, H. (2021). Study of Enhanced College Advising in Upward 
Bound: Impacts on Where and How Long Students Attend College: Appendix (NCEE 2022-002a). 
Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. 


This report is available on the Institute of Education Sciences website at http://ies.ed.gov/ncee. 


IES RZ Institute of 
amy Education Sciences 


NCEE 2022-002a 
U.S. DEPARTMENT OF EDUCATION 


Study of Enhanced College Advising in 
Upward Bound: Impacts on Where and 
How Long Students Attend College 


November 2021 


Tamara Linkow 
Amanda Parsad 
Abt Associates 


Alina Martinez 
Mathematica 


Hannah Miller 
Civilytics Consulting 


NCEE 2022-002a 
U.S. DEPARTMENT OF EDUCATION 


CONTENTS 


INTRODUCTION ..............ccccccccccccccccccccccccccccccccccccccccccccscceseccsccesccccccccsesccecccesccenscesscessccesccsccecccccccsccesccesccees 1 
SECTION A. ADDITIONAL DETAILS ABOUT FIND THE FIT ............cccccccscsccccccccccccccccccccccccccccccccccccccccseces 2 
A.1 Research-Based Strategies on Which Find the Fit Was Based ............ccecececececeeececececeeeeeeeeeseeeees 2 
A.2 Details about Find the Fit COMPOMENMS...............ecesececececececececetececececececececececeseeeeeeeeeeeeeeeteneeeees 3 
A.2.1 Personalized Student FolderS................eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeneaeeeaeeeaeeeeeaeaeeeneeee 4 
A.2.2.. -MeSSaging Program si; ciccscesdssceecsaiscsvasdasexedpahsaaveadasdeezsasadecboagiscaeddaesaxssaacdanvaaseasersagaaess 6 
A.2.3 Training Webinars for ACVISOIS.................::::eeeeeeeeeeeeeeeeeeeeeeeeeeeneeeeeeeeeeeeeaaeeeeeaeaeaeneeee 8 
A.3 Costs Of Find the Fite iccsccvicccstuccsactisvedecsvescavediavecensbaveeucivavcvgsviatevees ive sav cdiesuecta eevee teveesb votes 9 
SECTION B. ADDITIONAL INFORMATION ABOUT HOW THE STUDY WAS DESIGNED AND 
IMPLEMENTED ...............sccccccccccccccccccccccccccccccccccecccecccescceescceccecscecccccccccccccseccsecccesssesscesscossccesecees 10 
B.1. Conceptual Framework of Find the Fit ............ccccccecceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeaaeeeeaeaaaeeeeeeeaeeeeee 10 
Be2° “Study D@SIG0 és esaids dansesasis ean de danetstiveeradestietsaaedetuadisavvenn boca sasiede andes pando eea vies il 
B27, - Study Sample ice svcdss esevuscanvewes uacewsicdaan slang ui wantusdanvutusuawswatiis tasendudean Mavavavenbudeansndusuacenduanns 11 
B22; Randonr Assignment. « ccc2icicdesnsencanasdevnddantnanneatéceadcsesncdndesicrasdneddcenddarshanasadacenandeneanes 14 
Bi3 Data Collection ..i.c.cictececssiiaeac nieve dedendencdeda dees sdeeveessiiuuva clade vacesded gu eedatsdegesuueesa badaeabedaaansades 17 
B.3.1. Timing of Data Collection for the Study... ceeeeseseneneneneneeceneneneneneneneneneneneeeeeeeees 17 
B.3.2 Data Sources Used to Obtain Study Measures...........::ceeeseeeeeeeeeneeeneneeeeeeeeneneeeeeeneneees 17 
Bid. Analytic MethOdS visisi.css vane tiescsuseussvsuevscavcuduansbiueuuesvsi sane velveddassiuivadatsiedaaes budduuasvsntubsewentacaveudaas 19 
Bi4.1 «Study Measures’ s sce. iiiccdd dics ond cates saendiics eweeadanoddasvas peace ine ccwe sein ncednananenevaiennebsanas doecbianaed 20 
B.4.2 Estimating the Effectiveness of Find the Fit .........::::cccccccccccecccccececececececececeeeeeeeeeeeeeees 40 
BeA:3- “POwer-Amalyses esc sjcy_cciecebactadenscnnsicse ven vacilaslesadnudeacecdsatendis datneeuntenece tacenshantesnendsadeasenane 51 
SECTION C. SUPPLEMENTAL TABLES AND INFORMATION ON STUDY FINDINGS ..................ccescceee 53 
GA  Undlermiatchissssscsseccscdsdeecsccsatccsnteenesededenendedsauycedanguuc anasieaaededawradelesahanasgnepeesdengnanabonenacsdedgnanse 53 
C.2 Selectivity Level of the College Attended 0.000.000... eeeeeeeeeeeeeeeeeeeeeeeeeeseeeeeeeeeeeeeeeneneneneeeeeees 57 
C.2.1_ Selectivity Level of the College Attended in the Second Fall and Third Fall after 
HSH SCHOO]: 2.0.5.0 2 dave nniacevecvadesvecusacnsscdsdesvenusncevuctaderasecuednatenssechsesdedeateanseceyestadsaseess 64 
C.2.2 Alternative Definitions Of Quality 22.0... cececeeeeeeeceeeceeeeeeeseeeeeseeeeeeeseeeseeeeeeeeeneees 67 
GiB) SPOLSISTEN CEs ciicivnsaicteaastawiivenssidsassetwiun neu daaestawschcnissiedcascossasuwaseuusuncenesdasnsuwesusestisunnsusuevedsbeste 69 
C.4 Hypothesized Negative CONSCQUENCES..............::::seeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeaeeeeeeeaeaeeeaeneeea 72 
G41. (SHCKERPIICe i: 500d nastics astidadieidetastiedtasthetisssedialsdletetedvlatsliveleteiiedieriden lal 73 
GAD: INCU PLIGG isco: iece seta deatecede ce nese dha eceuedd yeetadave codecs nucisdcsuecudteetistielasvateste caberaueseateytieas 74 
C.4.2 Dropout or Transfer to a Less Selective College ...........cceeeeececececececececececececeeeeeeeeeeeees 76 
C.5 Familiarity of the College Attended .0......... cee eeeeeeeeeeneneneeeeeeeneneneneneeeneeensasecenececeeeceeeeeeeeeeees 77 
G.5.1. “Distance from HOmMe sje i 5so socssisncetaiecenciasinnessaivcessiavinnsseasteanestnstasandis Masi daivasaetivianecds 77 
C.5.2 Enrollment in Upward Bound Host Institution .............eceeeecececececececeeeceeeeeeeseeeees 79 
G.6:; ‘Relational Analyses jcc sesscevecvecs caesetiadutaivscuviudus weeds tuceseinanscauvinasadeluacs aeuasasa ema dndectsvasasatsinese 80 
SECTION D. EFFECTS BY LEVEL OF FIND THE FIT IMPLEMENTATION ...............ccccscccscccccccccccccccsccees 84 
REFERENCE G ...............cccccccccccsccccccccccccccccccccccecccesccesccscecccccceccccsccccecccescsceccesccescccccecececccsccssccsescceeccescceeoes 87 


List of Exhibits 


Exhibit A.1 


Exhibit A.2 
Exhibit A.3 
Exhibit A.4 
Exhibit B.1 
Exhibit B.2 
Exhibit B.3 


Exhibit B.4 
Exhibit B.5 
Exhibit B.6 
Exhibit B.7 
Exhibit B.8 


Exhibit B.9 


Exhibit B.10 


Exhibit B.11 
Exhibit B.12 
Exhibit B.13 
Exhibit B.14 
Exhibit B.15 
Exhibit B.16 
Exhibit B.18 


Exhibit B.19 


Description of Content of Each Find the Fit Component and Mapping to Challenges 


ACT OSSOC vi viiis cccisinedeuca iuciges ca suc lanea daecacessdelawes inva says ini ganvesnucanvsiuvasceaseuianessveuceenousconoaluodeacossuvanse: 5 
Saiiple Text MeSSages ass. ccciissssccasdsthaveisbencedves«dedecstascean deduadurst bvaseia vs vedebies told eevee dete iene. 7 
Find the Fit Programmed Messages..........::scsccccccecccccccccecececececenececenececeneceneececececececececeeeceeeeeeees 7 
Estimated Cost to Upward Bound Projects of Find the Fit ..........ccccececececeecececececececeeeeeeeeeeeeeenees 9 
Study’s Conceptual Framework of How Find the Fit Influences College SUccesS..............ccceeee08 11 
Characteristics of Study Projects versus All Eligible Upward Bound Projects ................:::::100+ 12 
Characteristics of Students in Study Projects versus Students in All Eligible Upward Bound 
PEOJOCUS vicceesccicccsvexcedesssbuccas secceccesvclbevs savcccuays Vivecalsstscdas davccccbaveeUedscavseeebsstsddassaudedessedsdauebaseviee’ 13 
Host Institution Type and Locale of Study Projects, by Random Assignment Status................. 15 
Characteristics of Study Projects before the Lottery, by Study Group...................eeeeeeeeeeeeeeeeees 15 
Characteristics of Students before the Lottery, by Study Group .................:e:eeeeeeeeeeeeeeeeeeeeeeeees 16 
Timing of Data Collection, Lottery, and Find the Fit Delivery ................::ceceeeeeeeeeeeeeeeeeeeeeeeeeeees 17 
Data Sources, Including the Sample, Timing of Data Collection, Response or Coverage 

Rate, and Content Used for Each ............ccccccsssscccceessececsneecesseneecessaneeccessaeecessageecessaneecessageeees 18 
Construction of Outcome Measures Based on Full Randomized Sample and Percentage of 
Students Missing Data.............ccccccccccsssssseseecececcasensseseecececcaseaeesseseececeseaeeasseseecececeasageeseeeceeeees 21 
Construction of Measures of College Characteristics and Percentage of Students Missing 

Dattiasssaaseide cites des saevsecd suncyastecsedeccias devia nisacmeckev cued ssusvencabacmadedatusdeces Wioeatnersiviassamsateretetmatsnanciees 26 
Comparison of College Enrollment Data Sources on Key Features ...........:.csscccccecceeeseseeeeeeeeees 28 
College Enrollment Rates by Data SOUICE ............. ee eee eeeeeeeeeeeeeeeeteteneeeeeeeneteteteteteneeeeeeeeeneaeeenee 29 
Overlap in College Enrollment across Data SOUILCES................:e:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeneeeeeeeeeeenee 30 
Congruence between Annual Performance Report and Baseline Student Survey Data............ 35 
Student and Project Characteristics Before the Lottery: Construction and Missing Data.......... 37 
Rationale for Exploring Effects for Subgroups of Students and Projects ..............eceseceeeeeeeeeees 39 


Characteristics of Students before the Lottery Used in the Analysis of Undermatch , by 


Characteristics of Students before the Lottery Used in the Analysis of College Entrance 
Exam Scores of Incoming Freshmen at College Attended, by Group ..............eeeeeseeeeeeeeeeeeeeeees 42 


Exhibit B.20 Characteristics of Students before the Lottery Used in the Analysis of Graduation Rate at 


Exhibit B.21 


College Attended, by Group ...............eeeeeeeeeeeeeeeeeteteteteeeteteneteneteneneneeeaeaeeeeeaeaeaeaaeaeaaeeeeeneeeaeee 43 


Characteristics of Students before the Lottery Used in the Analysis of Sticker Price at 
College Attended, Dy Group siyicessczvsessssvetessssieiescasencasdeaveleneaseelasesaseycnanssunadsscbencassasuiedeaeneuenes 44 


Exhibit B.22 


Exhibit B.23 


Exhibit B.24 
Exhibit B.25 
Exhibit C.1 
Exhibit C.2 
Exhibit C.3 
Exhibit C.4 


Exhibit C.5 


Exhibit C.6 
Exhibit C.7a 


Exhibit C.7b 


Exhibit C.8 


Exhibit C.9 


Exhibit C.10 


Exhibit C.11 


Exhibit C.12 


Exhibit C.13 


Exhibit C.14 


Exhibit C.15 


Exhibit C.16 


Characteristics of Students before the Lottery Used in the Analysis of Net Price at College 


Attended; by: Groupidecsccsg fee seciaedesthades tuceceeca stan hide caestuaideseevaseteaas Ae as eabibesas ceeeaseees aes 45 
Characteristics of Students before the Lottery Used in the Analysis of Distance from Home 

of College Attended, by Each Group ............cesesesececeeececececececececececececececeseseeeteseeeseeeeeeseeeeeeeeees 46 
Comparison of Design Assumptions and Sample Statistics for Undermatch................:csecseeeeees 51 
Achieved Minimum Detectable Effects for Overall Sample.................eeeeececececececececececececeeeeeees 52 
Effects on Undermatch, Overall and for SUDgrOUDS................eeesesesecececececececececececececeeeeeeeeeeees 53 
Sensitivity Analyses for Effects on Undermatch...............ecesesesesesecececececececececececececececeseeeseeeeeees 54 
Among Undermatched Students, Whether and Where Students Enrolled .................cccccceeeeees 55 
Differences in Undermatch Based on College Entrance Exam Scores of Incoming 

Freshmen, Overall and for SUDQTOUDS ..............esesesesececececececececececececececececececeseeeeetereeeeeeeeeeees 56 
Sensitivity Analyses for Difference in Undermatch Based on College Entrance Exam Scores 

Of Incoming Freshmen. ¢i.2.i..ccccceinceccsseneccasesngdecestngvceseiacseqedescbecslassvenavavevedssasgversvinevenstabeveves 57 
Effects on the Selectivity of the College Attended in the First Fall after High School................ 57 
Sensitivity Analyses for Effects on the Selectivity of the College Attended in the First Fall 

after High SCHOOL. «ssc. sisceccssensvesieiieescovangvcedeancecestang cacsavtbevesdvinevecadesevessvayensecavareversvaneveesdenevens> 58 
Benjamini-Hochberg correction for multiple comparisons in the enrollment domain 
(Undermatch and Selectivity Level) .............ccccsssssssseecececceceeessseecececeacenecsseeseececessauecseeseeeeeeeeas 59 
Effects on the Selectivity of the College Attended in the First Fall after High School, by 

Student’ Subgroup s.4.6.05.i tia aeeitiinae stearate diva nat all Wana 59 
Effects on the Selectivity of the College Attended in the First Fall after High School, by 
PFOjECtSUDSTOUP iii AAs Me aslais aidan ess vas leila nie iatltedes 62 
Analyses for Effects on the Selectivity of the College Attended in the Second Fall after High 
SCHOOL ss si:ccsss coescesds pevasayes coeces seecdan eit dau dive vsated edecvad hi sveuen dade spabavebegesdascheat is ooegentoeabaciaedesaie wae 65 
Analyses for Effects on the Selectivity of the College Attended in the Third Fall after High 
SCHOO]: ccdessstguecccdiedscssbiedededaieddassiagde Lecdivdeuestas vbavevidevegedebenleesidabsledes ued eeviosvedssebesiesbessvalasstapends 66 
Differences between College Entrance Exam Scores of Incoming Freshmen at the Colleges 
Attended by Find the Fit and Regular UB Advising Group Students, Overall and for 

SUDSTOUPS s3eicec2 svrisenisevethedvienteetieidecessceedenasevenths attvag aad elsegebea vi sav ehuabasevagh aaveluel aetateeaatnass 67 
Differences between Graduation Rates at the Colleges Attended by Find the Fit and 

Regular UB Advising Group Students, Overall and for Subgroups ...............eeeeseeeeeeececeeeeeeeeeees 68 
Effects on Continuous Enrollment into or Graduation by the Third Fall after High School, 
Overall -and for SUDSTOUDS sisics..ieiscescsssaseee sein capesssaueecsvanchecodeagseecnaasenenavancnenacedanenabancnacasanayens’ 70 
Sensitivity Analyses for Effects on Continuous Enrollment into or Graduation by the Third 

Fall. after High School i. :icsea.daceeented stis sacs adbecack Ded ase gens cach ada es vice Soa Wane bag haed Gand shouts dolea daa gee dua 71 
Impact of Find the Fit on Whether Students Enrolled and Persisted in College after High 

SChOO) iis. cei ieee iain aa iar ane bhai ia AG Anahi au Aen eee 71 


ili 


Exhibit C.17 Overall Effect and Sensitivity Analyses for Effects on Continuous Enrollment through the 


First Year after-Hish School: jic,3.ssiaiiic. cee heat ti diets ieee senate ala gave 72 
Exhibit C.18 Overall Effect and Sensitivity Analyses for Effects on Continuous Enrollment throughout 

the First Two Years after High SChOOI]...............ccccecccccecececececececececececececececeeceeecececececeeeeeeeeeees 72 
Exhibit C.19 Differences between Sticker Prices at Colleges Attended by Find the Fit and Regular UB 

Advising Group Students, Overall and for SUDgrOUPS ..............eeeeecececececececececececececececececeeneeees 73 
Exhibit C.20 Differences between Net Price at Colleges Attended by Find the Fit and Regular UB 

Advising Group Students, Overall and for SUDQrOUPS ..............eeeeececececececececececececececececeeeeeeeees 74 
Exhibit C.21 Effects on Dropout or Transfer to a Less Selective College, Overall and for Subgroups ........... 76 
Exhibit C.22 Sensitivity Analyses for Impact on Dropout or Transfer to a Less Selective College................. 77 


Exhibit C.23 Differences in Distance from Home of Colleges Attended by Find the Fit and Regular UB 


Advising Group Students, Overall and for SUDgrOUPS ...............ceeececececececececececececececececeeeceeees 78 
Exhibit C.24 Effects on Enrollment at Upward Bound Host Institution, Overall and for Subgroups............. 79 
Exhibit C.25 Sensitivity Analyses for Impact on Familiarity of the College Attended................0...eee 80 


Exhibit C.26 Persistence into the Third Fall after High School, by Selectivity of the College Attended in 
the First: Falli.ccscost srtessbevthetete esheets divlesstvdsels ennai ive Nees 81 


Exhibit C.27 Differences in Persistence into the Third Fall after High School, by Selectivity of the 
College: Attended in the: First: Fall). jscecccccsosessagvsceeaeadeseissag ees dunia uss ivnses ccevavava de vanasete save sen dandbeonses 81 


Exhibit C.28 Persistence into the Third Fall after High School, by Student Achievement and 
Underitiat ch: Status: :.¢: scsiacoitcatecsasc chiens thdesten Reese dss Rashae heades vadin std dieadewaleaskebncodaaeteues 83 


Exhibit C.29 Differences in Continuous Enrollment into the Third Fall after High School, by Whether 
Students Attended a Familiar College .............eceseseseseeececececececececececececececececeseseseeeseeeeeeeeeeeeeees 83 


Exhibit D.1 Effects on Undermatch, by Level of Find the Fit Implementation ........................::eeeeeeeeeeeeeeees 84 


Exhibit D.2 Effects on Selectivity Level of the College Attended, by Level of Find the Fit 
Implementation...s.20.cHedniiisi tii aie eeiiniaeevealiviedeaeisalesstaehavdetaeiene 84 


Exhibit D.3 Effects of Find the Fit on Persistence into the Third Fall after High School, by Level of Find 
thé Fit Implementati On cise scncasaezeesient iu saaclerede wai vuasadessaaeet utes get iensageic ands chewemnstecanaspeedenssaels 86 


iv 


INTRODUCTION 


This appendix is a companion to the Study of Enhanced College Advising in Upward Bound: Impacts on Where and 
How Long Students Attend College report. The appendix provides additional information about the Find the Fit 
enhanced advising strategies and the study that assessed its effectiveness when implemented within Upward 
Bound projects. This additional information includes details on how the study was designed and conducted, the 
statistics that support key findings in the report, and exploratory analyses performed to further investigate and 
understand those key findings. The content of the appendix is referenced throughout the main report. 


SECTION A. ADDITIONAL DETAILS ABOUT FIND THE FIT 


This section provides additional information about Find the Fit, including the research on which it was based, 
details about Find the Fit’s three components, and the cost of Find the Fit to implement. This detail is intended to 
allow others to use or build upon Find the Fit as tested in this study. 


A.1. Research-Based Strategies on Which Find the Fit Was Based 


Find the Fit brought together promising strategies for college advising identified by research that was emerging 
alongside increasing interest in reducing the number of students who “undermatch”—meaning they do not enroll 
in college or do not enroll in the most selective college they likely could attend. These strategies had undergone 
rigorous study, but the evidence on these strategies was still developing. Strategies from these studies were 
combined into Find the Fit. Although these strategies had been tested with populations and in settings somewhat 
different from Upward Bound—for example, with high-achieving, low-income students or students with limited 
access to college advising— they addressed some of the common challenges that students like those in Upward 
Bound face in finding and enrolling in a college that is a good fit. This section describes the research that 
informed the key strategies in Find the Fit and the adaptations that were made to suit the Upward Bound 
program model and population of students. 


Customized information about college opportunities, costs, and quality. In one study, low-income, high- 
achieving students who received customized information packets with information about applying to colleges, 
key milestones in the application process, and sample colleges to which they were admissible applied to and 
were admitted to more colleges, including more selective colleges that had higher graduation rates, than did 
students who had not received these packets.! The information sent to students was extensive, and the study 
specifically targeted high-achieving students who were not receiving much college advising and who would likely 
bear no cost if they attended a highly selective college because of generous financial aid packages. Find the Fit 
included student materials from that study, which were adapted to better reflect the diverse achievement and 
literacy levels of Upward Bound students and the likely costs they would face if enrolling at more selective 
colleges. Because of this variability, the similar Find the Fit content was tailored to the academic skills of each 
student, so that a student’s sheet showed colleges to which a student was likely admissible, along with the 
colleges’ graduation rates and average cost paid by low-income students. Further, because Upward Bound 
students already receive college advising through the program, Upward Bound advisors received guidance on 
how to integrate the student materials into their existing college advising. 


Short activities to reduce students’ fears of settling into and being successful in unfamiliar school 
environments. Psychologists have found that brief activities can reduce students’ fears about fitting in or 
belonging in unfamiliar situations such as an unknown college.” For example, students exposed to a two-part 
activity—first reading about other students who initially felt that they did not belong in college but whose sense of 
not belonging dissipated over time, and then writing about how their own worries had changed over time— 
earned higher grades and reported being happier than did students who were not exposed to the activity.* 
Students in that study were college freshmen. With input from one of the study’s principal investigators, Greg 
Walton, an activity was adapted for Find the Fit to instead target high school students looking ahead to college 
and perhaps feeling anxious about the transition to a new and unfamiliar setting. In Find the Fit, the activity 
involved hearing stories from college students, some of whom participated in the Upward Bound program, 
where they shared their stories of doubt and how they adapted to college. Upward Bound students then 
explored their own experiences where they had adapted and succeeded in unfamiliar setting or after initial 
setbacks. 


. Hoxby and Turner 2013. 
2 Aronson, Fried, and Good 2002; Yeager et al. 2014. 
. Walton and Cohen 2011. 


Semi-customized text messages about college logistics. Several previous studies suggested that sending 
students semi-customized text messages increased college enrollment, college persistence, and Free Application 
for Federal Student Aid (FAFSA) completion for some groups of students. The text message customization varied 
but often included the students’ name and information specific to the college they were attending or considering 
attending. Some of the students who received these messages were more likely to go to college or persist in 
college, particularly those enrolling or enrolled in two-year colleges.* Those students who benefitted might have 
been those who had less access to college-planning supports, or who were not far along with their college 
planning at the completion of high school. Find the Fit modified the text message content to be appropriate for 
the milestones that Upward Bound students would face in the college application process and to remind students 
about application deadlines, financial aid resources, Find the Fit materials, and key pre-enrollment steps for 
college, for example, reminders to register for orientation, and check health insurance options. 


Concrete guidance on actionable steps. Find the Fit also incorporated key messages to encourage students to: 


e Apply to four or more colleges. Research suggests that applying to more than one college significantly 
increases a student’s chance of actually enrolling as well as the selectivity of the college where the student 
enrolls.° Find the Fit emphasized applying to four or more colleges and applying to more selective colleges. It 
did this because at the start of the study, College Board recommended students apply to at least four colleges 
to ensure a good fit and because attending a more selective college is associated with a higher chance of 
graduating and a shorter time-to-degree as well as higher earnings. ° 


e Complete the FAFSA by early spring of their senior year. Completing the FAFSA early opens up the most 
opportunities for institutional and state aid.’ It also gives students accurate information about real costs 
before they must decide about which college to attend. 


e Examine graduation rates at different colleges to which they are likely admissible. Providing guidance on 
selecting colleges that have high graduation rates and are a match with students’ academic qualifications can 
increase students’ chances of completing a four-year degree. ® 


Previous research suggests that low-income students often have limited information about the importance of the 
steps above.° Providing students with this type of guidance can lead to higher rates of enrollment in selective, 
four-year colleges and in enrollment shifts from two-year to four-year colleges. '° 


A.2 Details about Find the Fit Components 


As described in the report, Find the Fit comprised three components, each of which had multiple elements. The 
components together aimed to address common challenges that low-income students face in finding and 
enrolling in a college that is a good fit for them. These challenges include financial hurdles, the logistics of 
applying to college, and limited expectations about the types of colleges that may be a good fit for students. 
Exhibit A.1 maps the Find the Fit components and the contents of each to these three key challenges. The section 
describes each of the components—personalized folders with student materials, text messages, and training 
webinars for advisors—in more detail. 


$ Castleman and Page 2015, 2016; Page, Castleman, and Meyer 2018. 

5 Pallais 2015; Smith 2013. 

The better outcomes for students at more selective colleges are likely due in part to the characteristics of students who attend these schools, including their 
stronger academic credentials, but could also be due to the institutional resources more selective colleges can provide (Bound, Lovenheim, and Turner 
2010; Bowen, Chingos, and McPherson 2009; Dillon and Smith 2020; Horn and Carroll 2006; Howell and Pender 2016; Hoxby 2001; Smith 2013; Witteveen 
and Attewell 2017). 


e Cannon and Goldrick-Rab 2016; Feeney and Heroff 2013. 
. Roderick et al. 2008. 

3 Bowen, Chingos, and McPherson 2009. 

a Avery 2013. 


A.2.1 Personalized Student Folders 


The Find the Fit personalized student folders contained 13 handouts, exercises, and activities personalized for 
each rising senior at Upward Bound projects that had access to Find the Fit (Exhibit A.1). Modeled on information 
folders previously evaluated," some of the materials were customized for each student based on the student’s 
academic preparation and geographic location. These customized materials included information on 
scholarships available in the student’s state and sample colleges in that state and nearby states to which the 
student was likely admissible based on his or her academic preparation and achievement. Assembled in 
personalized folders, the materials were mailed to the Upward Bound projects in June 2015, the summer after 
students’ junior year. Projects also received copies of a letter and parent-focused timeline that they could use to 
share Find the Fit information with parents. These materials were intended to be integrated into the college 
advising those Upward Bound projects already offered. It was anticipated that versions of some of the materials— 
for example, the application timeline and information on college application fee waivers—likely were available to 
many projects even before Find the Fit but they were included to connect other items in the folder. Thus for each 
of the materials, projects could decide whether to add it to the resources they used with students, replace an 
existing resource they previously used, or not use it. 


tt See Hoxby and Turner (2013). 


Exhibit A.1 


| Component 


Personalized Student Folders 


24 Messages sent to 


Students (about 
twice per month)? 


Description of Content of Each Find the Fit Component and Mapping to Challenges 


Addressed 


Content 


Shuffle, Sort, and Stack activity to prompt thinking about a variety of 
factors, and which students value, in considering colleges 

Four Factors of Fit handout to promote thinking about a variety of 
factors, including academic quality, in considering colleges 

College Application Timeline Reminders sheet of key steps in the college 
application process, including financial aid applications 

My College Planner booklet to track steps in the college application 
process 

My SCOOP-—Sample Cost, Outcomes, and OPportunities Sheet for 
College that includes a customized set of example colleges to which that 
student is admissible, to demonstrate the range in out-of-pocket costs, 
show variation in institutional quality, and counter misinformation about 
college costs 

Scholarships and Grants guide that emphasizes the importance of 
searching for a wide range of scholarships and grants to minimize 
financial burden 

My College Search tracking sheet to record key factors about colleges 
being considered 

Break Beyond the Familiar video and activity to encourage students to 
recognize their own ability to learn and grow in unfamiliar environments 
Discover Campus Support Services activity to identify support services 
and staff available at colleges of interest 

Start Your 2+2 Planning guide for planning transfer to a four-year college 
for those starting at a community college 

The Common Application information sheet to facilitate completion of 
the Common Application 

College Admission Application Fee Waivers information sheet about 
waivers of college application fees 

National Association for College Admission Counseling Request for 
Admission Application Fee Waiver form to facilitate submission of 
request for waiver of college application fee 

Real-time customized reminders to not miss key college application and 
pre-enrollment deadlines 

Prompts to complete the FAFSA early; links to resources to find 
scholarships 

Prompts to explore a variety of colleges 

Reminders to use Find the Fit materials 


Challenge(s) 
Addressed 


Limited Expectations 


Logistics of Applying, 
Limited Expectations 
Logistics of Applying, 
Financial Hurdles 
Logistics of Applying, 
Financial Hurdles 
Financial Hurdles 


Financial Hurdles 


Logistics of Applying, 


Limited Expectations 


Limited Expectations 


Limited Expectations 


Limited Expectations 


Logistics of Applying 


Financial Hurdles 


Limited Expectations 
Financial Hurdles 


Logistics of Applying 
Financial Hurdles 
Limited Expectations 


Logistics of Applying, 
Financial Hurdles, 


Limited Expectations | 


Component Content Challenge(s) 
Addressed 
oD Review of emerging research and promising practices, including Logistics of Applying, 
= : consequences of attending less selective colleges than students could Financial Hurdles, 
‘Ss | attend and benefits to enhancing students’ feelings of social belonging Limited Expectations 
= and beliefs that intelligence can be increased with effort 
| 2 SS Training on use of Find the Fit student materials in the folders and text Logistics of Applying, 
ao] messaging Financial Hurdles, 
* Limited Expectations 


a: Students projects with a summer bridge program were sent an additional five messages during the summer between high school and college. 


A.2.2 Messaging Program 


To remind participants in real time about important steps in planning, targeting, applying, selecting, and 
enrolling in college, Find the Fit included a series of programmed messages sent to Upward Bound students. The 
messages were adapted from an earlier study that showed personalized text message reminders sent to high 
school students could increase their college enrollment rates."* The messages reminded students about key steps 
in the college search, application, and enrollment processes. Messages were sent via a web-based text messaging 
platform provided by the vendor Signal Vine. Signal Vine’s platform allowed messages to be programmed and 
semi-customized and allowed advisors to send response messages to students via the platform. When students’ 
cell phone number was not available, students were sent the messages through email instead. About one-quarter 
of students received email messages for this reason. 


The messages began at the end of students’ junior year (the start of Find the Fit programming) and continued 
until the end of students’ senior year, or through the summer after high school for students in the 65 projects 
with access to Find the Fit whose Upward Bound programming extended through the summer. Messages were 
automatically sent out at important time points, such as prior to college application due dates. Students received 
about two programmed messages per month. 


Messages were customized to include each student’s first name, advisor’s name, and information specific to the 
student’s college plans. Customization was a key feature of the text messaging program because prior studies 
suggested that individuals are more likely to take action if the information they receive is relevant to them.” For 
example, students who provided a list of colleges to which they planned to apply, either through the initial 
student survey prior to the start of Find the Fit or in response to a fall programmed message asking them about 
their application plans, were automatically sent deadline reminders several weeks before each college’s 
application was due. To include the deadline reminders and other college-specific information, the study team 
manually looked up key information about each college students planned to attend. Exhibit A.2 illustrates some 
of the messages sent to students. Exhibit A.3 describes the full set of programmed messages, including when 
messages were sent, the messages’ focus, and the Find the Fit student materials mentioned in specific messages. 


. Castleman and Page 2015. 
- Castleman and Page 2015; Castleman, Schwartz, and Baum 2015; Karlan et al. 2010. 
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Exhibit A.2. Sample Text Messages 


Hi Sara! Do u have a list of colleges Remember - as an Upward Bound 
on ur My College Search sheet? Use student, u can waive many college 
it to compare other Fit Factors and application fees! Use this form 
narrow down ur list. Talk to ur http://bit.ly/ZSKdal. Talk to ur 
advisor if u need help. advisor if u have questions. 


Hi Camilla! Have u completed ur 
FAFSA? Do it before March 15 to get 
the most financial aid possible! 
https://fafsa.ed.gov/Txt ‘HELP’ if u 
need FAFSA help! 


Hi Darryl, just a reminder that the 
appl for CSU-Chico is due on 
11/30/2015! Talk to ur advisor if u need 
Jatc yo iventjevletomaeto-}o)e)iCe-tuloyels 


Exhibit A.3. Find the Fit Programmed Messages 


May 2015 Introductory message 
June 2015 Reminder to use college search tools (links to tools included) My College Search 
July 2015 Reminder to begin college application list My College Search 
August 2015 Encouragement to compare college costs after financial aid and to My SCOOP Sheet for 
research campus support services College; Discover 
Campus Support 
Services 
September 2015 Reminder to finalize college application list My College Search; My 
SCOOP Sheet for 
College 
October 2015 Request for students’ college application list 
November 2015 Reminders to use application fee waivers and to check whether The Common 
colleges accept the Common Application Application; College 
Admission Application 


Fee Waivers 
Winter 2015-16 Automatically timed application deadline reminders; 
generic deadline reminder on December 15 for students who 
did not provide a college application list 


January 2016 Reminder and link to complete the FAFSA 
February 2016 Resources to search for scholarships; second reminder to 


complete the FAFSA 

March 2016 Reminder to check college graduation rates 

April 2016 Offer to help interpret financial aid award letter; request for 
students to reply with which college they planned to attend in the 
fall 


Student Material(s) 


Date Sent Focus of Message Referenced 

May 2016 Second request for students’ college choice 

June 2016 Reminders to get in touch with campus support services and to Discover Campus 
log in to the college’s web portal to check key enrollment steps Support Services 
and deadlines | 

July 2016 Reminders to register for orientation, plan for first tuition | 
payment, register for placement tests, and check health insurance 

| options 
August 2016? Congratulation to students and good luck wishes 


a Sent in May 2016 to students in Upward Bound projects whose programming finished at the end of the school year rather than extending through the summer. 


A.2.3 Training Webinars for Advisors 


The study provided three live webinar trainings for Upward Bound advisors to educate them on the research 
underlying the Find the Fit materials and help them determine how to integrate these materials into their college 
advising. All staff who provided college advising to rising seniors in the Upward Bound projects with access to 
Find the Fit were encouraged to attend each webinar. Each webinar was about one to one-and-a-half hours and 
was offered at four different times in April or May 2015 so that advisors could attend at a date and time 
convenient for them. The webinars were also recorded and made available online. Each webinar, briefly 
described below, was led by a facilitator with extensive experience working to promote college access or 
improve educational success among low-income or minority students. 


Webinar 1 - Why We Want to Find the Fit. The goal of the first webinar was to support advisors in addressing 
students’ beliefs about college costs and emphasize the importance of attending a college that is a good match for 
students academically. This webinar provided an overview of the webinar series; described the tools and 
materials that Find the Fit was providing; explained that Find the Fit would enhance what advisors were already 
doing and that a focus on college “fit” and academic match could improve their students’ college enrollment and 
persistence rates; and demonstrated how advisors could use some of the Find the Fit student materials in the 
folders. 


Webinar 2 - Breaking beyond the Familiar: Empowering Students to Succeed in New Environments. 
The goal of the second webinar was to address students’ potentially limited expectations about where they could 
be successful at college. This webinar focused on how to recognize students’ beliefs or misconceptions that 
might affect the extent of their college search; how to use an activity shown to improve both students’ 
perceptions that they can be successful and their actual academic success in unfamiliar situations; and how to 
take advantage of other resources that can increase students’ comfort levels in applying to unfamiliar colleges. 
These resources include virtual campus tours and introductions to the federally funded Student Support Services 
programs at some colleges. 


Webinar 3 - Making It Manageable: Timing, Tips, and Tools to Meet Logistical Challenges. The goal of the 
final webinar was to provide advisors with additional tools to motivate students to go through the logistics of 
selecting and applying to a range of colleges. The webinar summarized research on why applying to more 
colleges increases students’ likelihood of enrolling; discussed important planning steps that students should take 
to make sure they stay on track throughout the entire application process; provided an overview of how text 
messaging can successfully nudge students to complete college application and enrollment tasks; and described 
the Find the Fit messaging program. 


A.3 Costs of Find the Fit 


For the study, Find the Fit was provided to Upward Bound projects at no cost. If Upward Bound projects chose to 
use Find the Fit in the future, the estimated cost to a project would be $13.50 per student. 


The study estimated the cost using the “ingredient method.” This method required first identifying all material 
and labor resources needed to carry out Find the Fit (Exhibit A.4). Then costs were attached to material 
resources, as those are the additional costs above and beyond regular Upward Bound programming costs. The 
Upward Bound program covers the labor costs of providing college advising to students. 


The estimate includes two main costs, both of which are material costs: printing student materials for the 
personalized folders and the messaging platform. Printing the 35 pages of student materials on a standard laser 
printer is estimated to cost $5.24 per student. The messaging platform used by the study, Signal Vine, offers 
different pricing options depending on the number of students and groups, such as high schools, that participate 
and on the level of message customization and support provided. The cost of accessing a messaging platform 
similar to the one for this study would be $8.25 per student. 


This cost estimate does not include costs associated with advisors’ time, such as attending the training webinars 
or responding to student messages, under the assumption that advisors could shift how they spend their time to 
incorporate these activities rather than others. Additionally, this estimate does not include costs related to 
generating the student materials that were customized for each student based on the student’s academic 
preparation and geographic location. Generating these materials requires looking up information for individual 
colleges, which students could do as part of their college search process. 


Exhibit A.4 Estimated Cost to Upward Bound Projects of Find the Fit 


WEIoM EUS 


Generate customized SCOOP sheets n/a n/a 
Generate customized scholarship and grants sheets n/a n/a 
Print student materials $131.00 $5.24 
Download advisor handbooks n/a n/a 
Access messaging platform $206.25 $8.25 
Watch webinars n/a n/a 
Explain materials to students n/a n/a 
Collect college data for messages n/a n/a 
ene monitoring, data processing, and program aa na 
maintenance for messaging 

TOTAL COST $337.25 $13.49 


a Based on assumed project size of 25 students. 


SECTION B. ADDITIONAL INFORMATION ABOUT HOW THE STUDY WAS 
DESIGNED AND IMPLEMENTED 


This section provides details about how the study was designed and carried out to address the study’s research 
questions. This detail is intended to document the study sample, data sources and measures, analytic methods, 
and the achieved power of the study to detect effects of Find the Fit. It should allow other researchers the 
opportunity to build on or replicate the study with other advising strategies or in other settings. 


The study was designed to answer the following three key questions: 


1. Did Find the Fit improve college going among Upward Bound students? That is, did it reduce college 
undermatch, increase the selectivity of the colleges students attended, or increase postsecondary 
progress? This central question informs whether Find the Fit could be a strategy for Upward Bound projects 
to integrate into their existing college advising to improve their students’ college outcomes. 


2. To what extent did Upward Bound projects implement Find the Fit, and how did that 
implementation affect the college advising received by their students?“ These findings provide insight 
into how projects adopted Find the Fit and the extent to which it changed their practices. 


3. Are there impacts of Find the Fit for some subgroups of Upward Bound students or projects and not 
others? This information could be useful to the program office in providing technical assistance to help 
Upward Bound projects improve their advising or to individual Upward Bound projects trying to determine 
whether adopting Find the Fit is appropriate for them. 


B.1. Conceptual Framework of Find the Fit 


The study was grounded in a conceptual framework of how Find the Fit would affect students’ college-going 
behaviors. These details are provided to enable readers to understand how the Find the Fit components, when 
projects choose to integrate them into their college advising activities, were expected to influence project 
implementation and students’ attitudes and behaviors (interim measures), which can serve as early indicators of 
college going (Exhibit B.1, below). The goal of Find the Fit is to reduce academic undermatch among Upward 
Bound students. Decreased undermatch, through increased college enrollment and enrollment at more selective 
colleges, is then expected to lead to higher rates of college persistence and completion. The study’s first report 
provides more details on how Find the Fit was implemented and affected students’ interim attitudes and 
behaviors. * 


- The study’s first report (Martinez et al. 2018) addressed this research question. 
8 Martinez et al. 2018. 
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Exhibit B.1. — Study’s Conceptual Framework of How Find the Fit Influences College Success 


FIND THE FIT INTERIM ATTITUDES 
COMPONENTS IMPLEMENTATION AND BEHAVIORS COLLEGE SUCCESS 


NateCoumuticourtc Advising on key topics Student attitudes Decreased college 

7 academic undermatch 
including increased 
college enrollment and 
college quality 


Net costs Importance of 

P (ero) | ferevemcwe: (ers (a (cont (a 
Academic match : E 
quality/reputation 
Academic outcomes 

of colleges, such as 

Text messages to graduation rates 

students Adaptive mindset 


ome eee. 5 - Increased college 
Application timelines Student behaviors 5 ai 


and key steps 


persistence and 
completion 


Apply to four or more 
fare) | (ereveny 


Apply to more selective 
feta) | (fevers 

Training webinars 
for advisors 


Advising on specific Complete FAFSA by 
behaviors March 15 


Apply to four or more 
colleges 


Complete FAFSA by 
NY FeV velo by 


Notes: Arrows show how Find the Fit’s components, if integrated by Upward Bound projects into their college advising activities, are expected to influence 
advising and interim outcomes and how the interim outcomes might influence the later outcomes. 


B.2. Study Design 


The study team recruited Upward Bound projects to participate in the study and randomly assigned projects 
either to offer the enhanced advising—Find the Fit—to high school seniors during the study period or to offer 
regular Upward Bound advising to seniors during the study period. This section describes how the study team 
recruited Upward Bound projects and students for the study and how projects were randomly assigned. 


B.2.1 Study Sample 


The study aimed to serve students who were 2015-16 high school seniors, in order to have sufficient time during 
the study period to follow participating students into and through some of their college experience. Therefore, 
the study focused on regular Upward Bound projects funded with fiscal year (FY) 2012 funds as these projects 
were expected to have their group of students reach senior year in high school in 2015-16. 


To identify eligible projects, the study used information maintained by the Department of Education on the 
Upward Bound awards granted in FY 2012. All 823 regular Upward Bound awards funded with FY 2012 funds 
were eligible to participate in the study. Awardees that shared staff or brought together students across awards 
to provide services were treated as single projects for this study. This ensured that all Upward Bound students 
served by the same staff or host institution would either have access to the study’s enhanced advising—Find the 
Fit—or not, minimizing the potential of spillover from the enhanced advising group to the regular advising group. 
Thus, the 823 awards formed 702 eligible Upward Bound projects. The study intended to recruit 200 of the 702 
projects (see Section B.4.3 for statistical analyses to determine the needed sample size). Ultimately, 194 projects 
volunteered to participate in the study. These 194 projects served 4,443 rising 2015-16 seniors." 


~ An additional 11 rising 2015-16 seniors served by the study projects were excluded from the study prior to random assignment because their parents did not 
grant consent for them to participate. Six of these students were in projects later assigned to the group receiving Find the Fit and five were in projects later 
assigned not to receive it. 
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Projects participating in the study were not randomly selected from the full set of eligible projects to statistically 
represent the Upward Bound program overall. However, the similarity of participating projects to all eligible 
Upward Bound projects suggests that results from the study could be used to inform decisions about whether to 
implement Find the Fit in other Upward Bound projects. Exhibit B.2 shows that projects in the study were similar 
to all eligible Upward Bound projects on important dimensions—they were primarily hosted by four-year colleges 
(57 percent), with almost half located in city settings (49 percent), and spanned all regions of the United States. 


Exhibit B.2. | Characteristics of Study Projects versus All Eligible Upward Bound Projects 


Host Institution Type 


Four-year college 56.7 58.5 
Two-year college 31.4 31.3 
Not a college (Other) 
Locale 
City 49.0 48.8 
Suburb 17.0 19.7 
Town 2D) 22.7 
Rural 11.9 8.7 
Northeast 14.4 14.5 
Midwest 27.3 22.5 
South 37.1 36.8 
West 19.6 23.6 
Other 1.5 2.6 
Yes 22.2 28.2 
Percentage of students who enrolled in college 85.1 85.0 
Project Size OY Corte) (Mean) 
Number of students 73.1 73.2 
TOTALN 194 702 


Sample Sizes: 
Host institution type, region, minority-serving host institution, and project size: Study = 194 projects, All = 702 projects. 
Locale: Study = 194 projects, All = 633 projects. 
Project historical college enrollment rate: Study = 186 projects, All = 672 projects. 

Source: APR 2014-15; IPEDS 2015-16. 


Likewise, students in the study projects who were rising 2015-16 seniors were similar to rising seniors in all 
eligible Upward Bound projects (Exhibit B.3). In both participating projects and all eligible projects, more than 
half of Upward Bound students were female (64 percent), they most commonly had a GPA of a B or better (71 
percent), and they were primarily students of color (about two-thirds Black, non-Hispanic or Hispanic). These 
similarities suggest that results based on students in the study sample are generalizable to the broader 
population of Upward Bound students. 


Exhibit B.3. Characteristics of Students in Study Projects versus Students in All Eligible Upward 
Bound Projects 


Female 

Hispanic 25.8 30.1 
White, non-Hispanic 23.4 21.4 
Black, non-Hispanic 38.6 37.4 
Other, non-Hispanic 12.2 11.1 
Low-income household 87.6 88.1 
First generation to college 91.7 92.2 


High School Course Taking 


Taken one or more AP/IB course 


Unweighted Cumulative GPA 


3.7-4.0 (mostly As) 18.4 16.5 
2.7-3.6 (mostly Bs) 52.4 55.3 
1.7-2.6 (mostly Cs) 26.2 25.5 
1.0-1.6 (mostly Ds) 2.8 2.5 


0.0-0.9 (mostly Fs) 


College Entrance Exam Score Quartile 


Highest quartile 

Second quartile 12.0 2 
Third quartile 18.6 2 
Lowest quartile 37.5 a 
Missing score 27.0 a 


Total N 4,443 18,487 


AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 

*College entrance exam scores were available only for students in study projects. 

Sample Size: Rising high school seniors in study projects = 4,443, Rising high school seniors in all eligible Upward Bound projects = 18,487. 
Source: APR 2012-13 to 2014-15; college entrance exam data 2015. 


B.2.2 Random Assignment 


The goal of random assignment was to create two study groups—projects providing Find the Fit and those 
providing regular Upward Bound (UB) advising—that were similar on characteristics likely to be related to college 
undermatch, enrollment, and persistence. That way, any differences that were seen in college-going behaviors 
between the two groups could be attributed to Find the Fit rather than to initial differences between the groups. 


During the study period, students in both Find the Fit and regular UB advising projects continued to receive 
Upward Bound’s existing services that could include: 


e Academic tutoring and instruction to prepare students to complete secondary or postsecondary courses. 
e Guidance on high school course selection. 

e College advising. 

e Assistance in preparing for college entrance exams such as the SAT and ACT. 


e Information on all federal student financial aid programs and benefits, as well as resources for locating 
public and private scholarships. 


e Assistance in completing college admission and financial aid applications. 


e Education or counseling services to improve the financial and economic literacy of students and their 
parents, including financial planning for postsecondary education. 


After the 194 Upward Bound projects and their 4,443 rising 2015-16 seniors had been identified and recruited, the 
projects were randomly assigned though a lottery to receive Find the Fit. Projects had a 50 percent chance of 
being assigned to either group-that is, for every project assigned to provide Find the Fit and regular UB advising, 
a project was assigned to provide regular UB advising. As part of the recruitment strategy, projects that 
volunteered were all promised the opportunity to receive Find the Fit, if not right away, as an incentive to 
volunteer. Projects randomly assigned to the Fit the Fit group received it during the study. Projects randomly 
assigned to the regular UB advising group got access to Find the Fit only after the 2015-16 seniors had left Upward 
Bound projects and the study had concluded. Thus, there is little possibility that students in the regular UB 
advising group experienced Find the Fit during the study period (that is, it is unlikely that the regular UB advising 
group was “contaminated” by access to Find the Fit advising). 


To both prevent a bad draw by chance (for example, more Find the Fit projects in urban settings than were 
regular UB advising projects) and to enhance the study’s ability to examine impacts of Find the Fit for key 
groups” of Upward Bound projects, projects were divided into eight groups (“randomization blocks”) created by 
the combination of their host institution type (four-year or not) and their geographic locale (city, suburb, town, 
or rural) for random assignment. Within each block, half the projects were randomly assigned to the Find the Fit 
group and the other half to the regular UB advising group. Exhibit B.4 shows the number of Find the Fit and 
regular UB advising projects in each randomization block. Of the 194 study projects, 98 were randomly assigned 
to the Find the Fit group and 96 to the regular UB advising group. 


- See Exhibit B.16 below for details on the rationale for including these measures as key subgroups in the study. 
me Some of the randomization blocks had an odd number of projects to be assigned to each group, this resulted in two more projects being assigned to the 
Find the Fit group than the Regular UB advising group overall. 
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Exhibit B.4 Host Institution Type and Locale of Study Projects, by Random Assignment Status 


Four-year host institution/City 29 29 58 
Four-year host institution/Suburb 10 9 19 
Four-year host institution/Town 13 13 26 
Four-year host institution/Rural 3 4 7 
Non-four-year host institution/City 20 17 37 
Non-four-year host institution/Suburb 7 7 14 
Non-four-year host institution/Town 8 9 17 
Non-four-year host institution/Rural 8 8 16 

Total N 98 96 194 


Sample Size: Find the Fit group = 98 projects, Regular UB advising group = 96 projects. 
Source: IPEDS 2015-16. 


As expected, the random assignment procedures resulted in Find the Fit and regular UB advising groups 
comprising projects with similar characteristics measured before the lottery (or at “baseline”) (Exhibit B.5). 


Exhibit B.5 Characteristics of Study Projects before the Lottery, by Study Group 


Host Institution Type 


Four-year college 56.1 57.3 =1:2 .870 
Two-year college 31.6 31.3 0.4 £955 
Other 12.2 11.5 
City 50.0 47.9 
Suburb 17.3 16.7 0.7 .900 
Town 21.4 22.9 -1.5 .804 
Rural 11.2 12.5 -1.3 
Northeast 14.3 14.6 -0.3 .953 
Midwest 25.5 29.2 -3.7 .570 
South 36.7 37.5 -0.8 913 
West 22.4 16.7 5.8 313 
Other -1.1 

26.5 
Percentage of students 84.9 
who enrolled in college 
Project Size (Mean) LY Corte) 
Number of students 76.0 70.1 5.9 .089 


® p-Values are for tests of whether there was a statistically significant baseline difference in the characteristic in the row. 

Sample Sizes: Host institution type, locale, region, minority-serving host institution, and project size: Find the Fit group = 98 projects, Regular UB advising group = 
96 projects. Project historical college enrollment rate: Find the Fit group = 93 projects, Regular UB advising group = 93 projects. 

Source: IPEDS 2015-16. 
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Across the 194 projects, 4,443 students participated in the study: 2,336 students from Find the Fit projects and 
2,107 students from regular UB advising projects. As expected, the students in the Find the Fit and regular UB 
advising groups had similar characteristics before the lottery (Exhibit B.6). Thus, any differences in undermatch, 
college selectivity, and college persistence can be attributed to Find the Fit rather than to initial differences 
between the groups. 


Additional information on the similarities (or “baseline equivalence”) of the samples used to assess the effect of 
Find the Fit on other measures of students’ college going is presented in Section B.4.2. 


Exhibit B.6 Characteristics of Students before the Lottery, by Study Group 


Female 64.4 63.8 0.6 721 
Race/Ethnicity 

Hispanic 27.5 23.0 4.5 310 
White, non-Hispanic 21.9 24.9 -3.0 .460 
Black, non-Hispanic 38.6 39.0 -0.4 .940 
Other, non-Hispanic 12.0 13.1 -1.1 .695 
Low-income household 88.0 87.5 0.4 .744 
First generation to college 91.2 92.2 -1.0 363 
patie ee 36.6 31.4 5.2 156 
ae aoe cumulative GPA 31 30 0.0 571 
peeks enuance cram eet 874.3 862.4 11.8 323 
Planned to apply to college 90.1 88.1 2.0 081 
ae . eit cee 19.2 21.6 -2.4 213 


AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 
a p-Values shown in this column are for tests of whether there was a statistically significant baseline difference in the characteristic in the row. 
Notes: The Find the Fit group percentage and estimated difference are adjusted for the blocked random assignment design and the clustering of students within 
Upward Bound projects. 
Sample Sizes: 

Gender: Find the Fit group = 2,318 students, Regular UB advising group = 2,102 students. 

Race/ethnicity: Find the Fit group = 2,311 students, Regular UB advising group = 2,099 students. 

Low-income household: Find the Fit group = 2,310 students, Regular UB advising group = 2,088 students. 

First generation to college: Find the Fit group = 2,317 students, Regular UB advising group = 2,101 students. 

AP/IB course taking: Find the Fit group = 2,305 students, Regular UB advising group = 2,085 students. 

GPA: Find the Fit group = 1,969 students, Regular UB advising group = 1,855 students. 

College entrance exam score: Find the Fit group = 1,745 students, Regular UB advising group = 1,499 students 

Planned to apply to college: Find the Fit group = 1,882 students, Regular UB advising group = 1,701 students 

Undermatched at all colleges where planned to apply: Find the Fit group = 1,840 students, Regular UB advising group = 1,669 students. 
Source: APR 2012-13 to 2014-15; college entrance exam score data 2015; baseline student survey 2015. 


16 


B.3. Data Collection 


The study team collected data from several sources to assess the effects of Find the Fit and describe its 
implementation. This section presents the timing of data collection and then details about the data sources used 


to create the study measures. 


B.3.1 Timing of Data Collection for the Study 


The study began following students in spring 2015 at the end of their junior year of high school, at which time 
they completed the study baseline student survey. The study followed students through their senior year in 2015- 
16, during which time the Find the Fit group projects had access to Find the Fit. The study then continued 
following students into college. College-going measures for this report are observed the first year after students’ 
expected high school graduation (2016-17) through fall 2018, the start of students’ expected third year in college. 
Exhibit B.7 illustrates the timing of data collection relative to the timing of Find the Fit delivery and key education 
milestones such as students’ junior and senior years of high school and their first three years of college. 


Exhibit B.7 Timing of Data Collection, Lottery, and Find the Fit Delivery 


eS) Data Collection Activities 


Student rosters 


Immediate college Continuous Continuous enrollment, 
enrollment, enrollment selectivity 


undermatch, 
Initial student Follow-up student selectivity, other college College Continuous 
survey + project characteristics selectivity enrollment 


survey 


survey 


Junior Year of Senior Year of First Year Second Year 
High School High School Co) a Oe) | Carer of College 


Winter Spring Summer Fall Winter Spring Summer Fall Winter Spring Summer Fall Winter Spring Summer 


Find the Fit (treatment)? 


Lottery assigning projects access to Find the Fit or not 


B.3.2 Data Sources Used to Obtain Study Measures 


Data came from three surveys conducted for this study, as well as from administrative data maintained by the 
Department of Education and other sources. Exhibit B.8 provides detailed information on each of the study’s 


data sources. 
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Exhibit B.8 Data Sources, Including the Sample, Timing of Data Collection, Response or Coverage 


Rate, and Content Used for Each 


Study 
Surveys? 


Baseline student 
survey (pre-Find 
the Fit) 


Follow-up 
student survey 


Project survey” 


Administrative 


and National 


4,443 
students 


4,443 
students 


194 
project 
directors 


Spring 2015 


Late Spring 
2016 


Spring 2016 


Find the 

Overall Fit Group 
80.6% 80.6% 
(3,583) (1882) 
81.7% 82.2% 
(3,630) (1920) 
94.8% 95.9% 
(184) (94) 


80.7% 
(1701) 


81.2% 
(1710) 


93.8% 
(90) 


Regular 
UB 
Advising 


Students’ college-going expectations 
and plans; demographic 
characteristics 

College advising that students 
received; number of applications 
submitted; use of Find the Fit 
materials (Find the Fit group only); 
colleges to which students applied 


Features of the college advising 
offered to students over the study 
period; use of Find the Fit materials 
(Find the Fit group only) 


ELEY 


Student rosters 


Upward Bound 
Annual 
Performance 
Reports (APRs) 


Integrated 

Postsecondary 
Education Data 
System (IPEDS) 


NCES-Barron’s 
Admissions 
Competitiveness 
Index 


Education 
Longitudinal 
Study (ELS) of 
2002 


National Student 
Clearinghouse 
(NSC) 


194 
projects 


194 


projects 


1,158 
colleges‘ 


1,158 
colleges‘ 


15,000+ 
students 


4,443 
students 


Winter 2014 


School Year 
2014-15 


School Year 
2015-16 


Selectivity 
Ratings for 
2014 


Spring 2004 


NSC: Fall 
2016, 2017, 
2018 


Find the 
(0) 1) My ge cody) 
100% 100% 
(194) (98) 
100% 100% 
(194) (98) 
100% 100% 
(1,158) (772) 
56.5% 61.5% 
(654)4 (772) 
N/A® N/A 
N/Af N/A 


Group 
100% 
(96) 


100% 
(96) 


100% 
(698) 


59.5% 
(698) 


N/A 


N/A 


Records of students who would be 
rising seniors in the participating 
Upward Bound projects 

Reports submitted by each Upward 
Bound project with information on 
projects and participating student 
characteristics 


Data on college characteristics 


Classifications of four-year U.S. 
colleges by relative competitiveness 
of admissions 


Academic information for a 
nationally representative sample of 
high school students 


Student-level information on college 
enrollment 
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Federal Student 


4,443 Fall 2016, N/A® N/A N/A Student-level information on receipt 
Aid (FSA) data students 2017, 2018, of federal student aid (such as Pell 
2019 grants, Federal Work-Study) 

documenting college enrollment, 
college to where which federal aid 
was sent, and FAFSA completion 

College Board 4,443 School Year 73.0% 74.7% 71/1% Records of student scores on the SAT, 

and ACT college students 2013-14 (3,244) ACT, PSAT, and PLAN prior to the 

entrance exam School Year start of Find the Fit June 2015) 

data 2014-15 


Find the 
Overall Fit Group 
Program 94 Find School Year 100% (94 100%(94 NA Records of advisors’ attendance in 
monitoring data __ the Fit 2015-16 Findthe Find the training webinars, and analytics from 
projects; Fit Fit the texting platform on messages sent 
2,336 projects, projects, to and received by students 
students 2,336 2,336 
students) students) 


4 The study survey instruments can be found at: https://www.reginfo.gov/public/do/PRAICList?ref_nbr=201508-1850-001 

>Project directors were asked to complete the survey themselves or assign it to the staff person who was most familiar with their project’s advising. About 63 
percent of project surveys were completed by project directors. 

© Data were collected for all colleges students attended in the fall of the first, second, and third years after high school, which were identified in either the 
National Student Clearinghouse or in the Federal Student Aid database. The IPEDS data were used to determine characteristics of the colleges attended, including 
cost. The Barron’s Admissions Competitiveness Index data were used to determine the selectivity level of the college attended and academic undermatch. Of the 
1,158 colleges students attended, there were 312 colleges attended by students in both Find the Fit and Regular UB Advising groups, 460 colleges attended only by 
students in the Find the Fit group, and 386 colleges attended only by students in the Regular UB Advising group. The total number of colleges attended by 
students in the Find the Fit group was 772 (312 + 460). The total number of colleges attended by students in the Regular UB Advising group was 698 (312 + 386). 

4 Percentage and number of colleges found in the Barron’s Admissions Competitiveness Index database. The Barron’s selectivity level was able to be classified for 
the 56.4% of colleges found in the database. The selectivity of an additional 43.5% of colleges was able to be classified as either a four-year college (125) or a two- 
year college (379). Less than 1 percent (0.1%) of students attended colleges that could not be found in either the Barron’s database or IPEDS. 

© A response or coverage rate for the ELS is not applicable because data from the ELS were not matched to the students in the study sample. Instead, data from 
the ELS were used to predict the selectivity level of colleges to which students were admissible based on their academic credentials. The ELS prediction model 
was then applied to students in the study sample. 

Of the 4,443 students in the study sample, enrollment records were found in both NSC and FSA for 50.1% of students (2,226), in NSC only for 10.5% of students 
(466), and in FSA only for 11.4% of students (507). Students who do not have any enrollment record (28.0% or 1,244 students) are classified as not having achieved 
the outcome-—for example, not having enrolled in college. Thus, they do not have missing values for variables constructed from these data sources. 

5 Of the 4,443 students in the study sample, FAFSA submission records were found for 3,799 (85.5%). Students who do not have a FAFSA submission record are 
classified as not having completed the FAFSA. Thus, they do not have missing values for this outcome. 


B.4 Analytic Methods 


The study created measures about students’ college going and estimated the effectiveness of Find the Fit on these 
measures of college going for all students and for particular groups of students and projects (“subgroups”). This 
section describes the approach for examining the effects of Find the Fit. First, the section describes the study 
measures, the rationale for selecting each measure, and how the measures were constructed. Then the section 
provides details on the study’s analytic methods, including the samples and methods used to estimate effects of 
Find the Fit on students’ college-going measures. Finally, the study’s achieved power to detect effects of Find the 
Fit for each measure is provided. 
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B.4.1 Study Measures 


Student College-Going Measures 


The study examined the effects of Find the Fit on multiple measures of college going (see Exhibit B.9 below for 
details about the rationale for each measure). Study measures most directly related to the hypothesized effects of 
Find the Fit are discussed in the report and are the main or confirmatory outcome measures. These include (1) 
college academic undermatch, (2) the selectivity level of the college attended in the year after high school, and 
(3) persistence into the third fall after high school or graduation by that time. 


Exhibit B.9 defines each main outcome measure, including the rationale for selecting the measure, how it was 
constructed, and missing data rates. All of these measures were constructed for the full sample of students, and 
there were no missing data for any of the main outcome measures except college academic undermatch. Data on 
college academic undermatch was missing for students who were missing data on their academic credentials. 
That information was necessary to predict the highest selectivity of college to which they could be admissible, a 
key component of calculating undermatch. 


In addition, the study examined effects of Find the Fit on exploratory measures, related to the selectivity level of 
the college attended in intervening years, persistence in years prior to the third fall after high school, and other 

characteristics of the colleges students attended. These exploratory measures are described in Exhibit B.9, with 

results shown in Section C. 
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Exhibit B.9 Construction of Outcome Measures Based on Full Randomized Sample and Percentage of Students Missing Data 


College Undermatch 
Enrolment 


Main/ measure Find the Fit 
encouraged students to change 
their college application 
behaviors to reduce academic 
undermatch, which could be 
linked to longer-term outcomes 
such as persistence, completion, 
and earnings. 


NSC, FSA, 
NCES-Barron’s 
Admissions 
Competitiveness 
Index, IPEDS, 
ELS, APRs, 
College Board 
and ACT 


1 = The student was not enrolled in 
college on Oct. 1, 2016, or the college 
the student enrolled in had a selectivity 
level lower than the highest selectivity 
level to which the student had a high 
probability of admission given her/his 
academic credentials 

O = The student was enrolled (on Oct. 1, 
2016) at a college with a selectivity level 
equal to or higher than the highest 
selectivity level to which the student 
had a high probability of admission 
given her/his academic credentials 


2.7 5.3 3.9 
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College Selectivity 

Enrollment level of 
college that 
student 
attended 
(constructed 
for each year 
after high 
school 
Year 1: 2016 
Year 2: 2017 
Year 3: 2018) 


Find the Fit encouraged students 
to attend the college with the 
highest selectivity level to which 
they have access. Selectivity may 
be linked to college persistence 
and completion as well as 
earnings, and as such is a 
measure of quality. 

Main/ measure Barron’s 
selectivity ranking of college that 
student attended in year after 
high school (Oct. 1, 2016) 


Exploratory measure The 
study also examined this 


outcome in the second and third 
years after high school to 
determine whether Find the Fit 
had an ongoing impact on the 
selectivity of the colleges that 
students attended (Oct. 1, 2017; 
Oct. 1, 2018) 


NSC, FSA, 
NCES-Barron’s 
Admissions 
Competitiveness 
Index, IPEDS 


For each of the three years after high 
school (Oct. 1, 2016, 2017, and 2018), 
seven variables were constructed 
using each Barron’s selectivity 
level’ and two-year colleges: most 
competitive, highly competitive, very 
competitive, competitive, somewhat 
competitive, any four-year college, or 
any college (two- or four-year) 

1= The college where the student was 
enrolled on Oct. 1 had a given 
selectivity level or higher 

O = The student was not enrolled in 
college, or the college the student 
attended had a lower selectivity level 


0.0 0.0 0.0 
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in College 


NA 


Progressing Persistence 


Familiarity 
of the 
college 
attended 
(host 
institution) 


Find the Fit encouraged students 
to attend the college with the 
highest selectivity level to which 
they have access, with the goal of 
increasing persistence and 
completion. 

Main measure Continuous 
college enrollment into the third 
fall after high school (Oct. 1, 2018) 
or graduation by that time 


Exploratory measure The 
study also examined continuous 
college enrollment through the 
first year and the second year 
after high school or graduation 
by each time point 


Exploratory measure Find the 
Fit encouraged students to 
consider attending less-familiar 
colleges, beyond the Upward 
Bound host institution, to 
improve college selectivity and 
reduce undermatch. 


NSC, FSA 


NSC, FSA, APRs 


A measure of continuous college 0.0 0.0 
enrollment or graduation was 
constructed for three time periods: 
through the first year after high school 
July 1, 2016-June 30, 2017), through 
the first two years after high school 
July 1, 2016-June 30, 2018), and into 
the third fall after high school July 1, 
2016-Oct. 1, 2018) 

1= The student was enrolled in college 
throughout the given time period, with 
no break in enrollment of more than 5 
consecutive months, or was enrolled 
on Oct. 1, 2016, and graduated from a 
two- or four-year college during the 
period 

O = The student was not continuously 
enrolled in college and did not 
graduate from a two- or four-year 
college during the period 


1 = Student was enrolled in Upward 0.0 0.0 
Bound host institution in the fall after 

high school (Oct. 1, 2016) 

O = Student was enrolled in a college 

other than the Upward Bound host 

institution, or was not enrolled in 

college in the fall after high school 

(Oct. 1, 2016) 


0.0 


0.0 
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NA Transferto Exploratory measure The NSC, FSA, 1=Transferred from a more selective 0.0 0.0 0.0 


a less study explored whether there NCES-Barron’s _ college (where enrolled Oct. 1, 2016) to 
selective were any unintended Admissions a less selective college or dropped out 
collegeor consequences of encouraging Competitiveness by start of Year 2 (Oct. 1, 2017) 


drop out by — students to attend more selective Index 
thesecond colleges. Specifically, students 
fall after who enroll in more selective 
high school colleges might have difficulty 
meeting academic requirements 
and consequently might transfer 
to less selective colleges or drop 
out. 


O = Did not transfer from a more 
selective college to a less selective one 
or drop out by start of Year 2 (includes 
students who did not enroll in college 
initially after high school) 


*Domains are provided for confirmatory measures and are taken from the WWC review protocol for studies of interventions to support postsecondary success version 4.0, downloaded from 
https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/WWC supporting postsecondary success protocol 4.0 08-01-19 sxf.pdf. NA indicates that there is no relevant domain for these outcomes in any of the 
WWC review protocols. As these outcomes were designated exploratory and the study was not planning on doing any adjustments for multiple hypothesis testing a domain was not named. 

» As of 2014, Barron’s six selectivity rankings for four-year colleges were most competitive, highly competitive, very competitive, competitive, somewhat competitive, and noncompetitive, plus specialty/missing (not ranked). 
For this study, unranked four-year colleges were combined with noncompetitive colleges and considered more selective than two-year colleges but less selective than four-year colleges ranked as somewhat competitive. 
The study considered two-year colleges the least competitive institution type. 

Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018. 
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Measures of Other College Characteristics 


The study also examined other characteristics of colleges that students attended to determine whether Find the 
Fit shifted the types of colleges that students attended in ways that might be positive for students or might bring 
unintended negative consequences. The characteristics are (1) college entrance exam scores of incoming 
freshmen, (2) graduation rate, (3) sticker price, (4) net price, and (5) distance from home. 


Because the Find the Fit did not have an impact on college enrollment and to ease interpretation of the results, 
the analyses for these five college characteristics include only students who attended college in fall 2016. 
Including students who did not enroll in college in these analyses would depress the means or percentages, 
making the data difficult to interpret. For example, including students who did not enroll in college in the 
measure of distance from home would reduce the average miles. As a result, these analyses fall outside of the 
lottery framework, because they exclude students who did not enroll in college. Section B.4.2 documents 
baseline equivalence for the samples used to analyze differences in the characteristics of the colleges attended by 
students in Find the Fit projects versus regular UB advising projects for these measures because we are outside 
the study’s lottery framework. 


Exhibit B.10 below provides information on how each measure was constructed, as well as the rate of missing 
data for each among students who were enrolled in college. Missing data occurred because some colleges did not 
report data on some characteristics in IPEDS. 
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Exhibit B.10 Construction of Measures of College Characteristics and Percentage of Students Missing Data 


College 
entrance exam 
scores of 
incoming 
freshmen 
Graduation rate 


Sticker price 


Exploratory measure. The academic 
qualifications of students who choose to 
attend a given college, may affect the 
college’s level of instruction, a measure of 
quality. 

Exploratory measure. A college’s 
graduation rate provides another partial 
measure of quality because it reflects the 
instruction and supports provided by the 
college to help students graduate while 
also reflecting the characteristics of 
students who attend the college. 


Exploratory measure. The study also 
examined college costs since a potential 
negative consequence of attending more 
selective colleges could be higher costs. 
More selective colleges typically have a 
higher average posted cost (sometimes 
called “sticker price”) than less selective 
colleges do. 


NSC, FSA, 
College Board 


NSC, FSA, IPEDS 


NSC, FSA, IPEDS 


75" percentile of SAT scores for 
first-time, degree-seeking students 
at the college the student attended 
the first year after high school (Oct. 
1, 2016)* 

Percentage of first-time, full-time, 0.1 0.3 
degree-seeking students who 

graduated within 150 percent of 

“normal” time to completion? at the 

college the student attended the 

first year after high school (Oct. 1, 

2016) 

Posted cost of attendance, including 0.7 0.4 
tuition, fees, books, supplies, and 

living expenses, at the college the 

student attended the first year after 

high school (Oct. 1, 2016)* 


30.4 39.6 


34.8 


0.2 


0.3 
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Net price 


Distance from 
home 


Exploratory measure. More selective 
colleges often have more resources and 
are able to keep students’ costs down by 
making grants and scholarships available 
to low-income students such as those in 
Upward Bound. As a result, college 
selectivity does not necessarily 
correspond to higher out-of-pocket costs, 
or “net price.” 


Exploratory measure. Because local and 
familiar colleges are often the default for 
low-income and first-generation college 
students, Find the Fit encouraged students 
to consider attending less-familiar 
colleges. One way the study measured this 
was as the average distance of students’ 
college from the Upward Bound host 
institution, because students’ home 
addresses were not available. 


NSC, FSA, IPEDS 


NSC, FSA, APRs 


Average annual cost of attendance 0.1 0.3 
for students with household 

incomes of $30,000 or less after 

taking into account scholarships and 

grants at the college the student 

attended the first year after high 

school (Oct. 1, 2016)4 


Miles between the college the 0.0 0.1 
student attended the first year after 

high school (Oct. 1, 2016) and the 

student’s Upward Bound host 

institution 


0.2 


<0.1 


a SAT verbal and math scores were summed. This characteristic has more missing data than the other college characteristics because most two-year colleges do not require students to submit standardized test scores, 
and some four-year colleges—particularly specialty schools and noncompetitive colleges—also do not require standardized test scores. 

b “Normal” time to completion is within six years of enrollment at four-year institutions or within three years of enrollment at two-year institutions. 
c This is the posted cost of attendance for the full academic year for first-time, full-time students. In-state or out-of-state tuition was used depending on whether or not the student attended a college in the same state as 
his/her Upward Bound host institution. 
d This is the average annual cost for first-time, full-time in-state students who receive federal financial aid and have household incomes of $30,000 or less. 
Sample Size: Find the Fit group = 1,683 students enrolled in college, Regular UB advising group = 1,516 students enrolled in college. 


Source: IPEDS 2015-16. 
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Additional Information about Measuring College Enrollment 


All of the measures in this report require first determining whether students attended college during the fall 


semester after high school, and if so, where they attended college. There were three potential sources of college 


enrollment data for this study—the National Student Clearinghouse (NSC); federal Financial Student Aid data 


(FSA); and Upward Bound Annual Performance Reports (APR) data, which are submitted by each Upward Bound 


project to the Department of Education and contained at least some data for every student who entered the UB 


projects. 


Each of these data source covers overlapping, yet slightly different, groups of students. How and from where 
data are collected and released account for some of the differences in whether students appear as enrolled 
across the three sources. Specifically, student coverage, the matching algorithms, and the time period covered 
likely account for most differences (Exhibit B.11). Incomplete college coverage likely accounts for a very small 
amount of the discrepancies across the NSC and FSA data because the majority of students in the United States 
are enrolled in colleges that both participate in the NSC and accept federal financial aid funds. 


Exhibit B.11 Comparison of College Enrollment Data Sources on Key Features 
Feature NSC FSA APR 
Restricted to YES YES NO 
particular colleges _—_ Colleges sign up to provide Only colleges that accept Any college could be 
data to NSC (3,636 federal financial aid funds— listed 
postsecondary institutions that is, Title IV institutions 
participate?) (approximately 6,502 
institutions?) 
Excludes certain YES YES YES | 
students Only students from Only records for students Only Upward Bound | 
participating colleges (97.8% of receiving federal financial aid students are 
students enrolled in U.S. included 
colleges covered‘) | 
Students can YES NO NO 


prohibit the release 
of their information 


| Matching process is 
based on SSN 


Reporting is based 
ona 12-month 
academic period 


NSC suppresses data when 
students request that their 
data be blocked under FERPA‘ 
NO 

Primarily student name and 
date of birth* 


YES 

Specific enrollment dates allow 
for adjustable enrollment 
periods 


SSN is Social Security number. 


* National Student Clearinghouse 2021. 
In 2017-18, a total of 6,502 Title IV two-year and four-year institutions in the United States and other U.S jurisdictions participated in Integrated Postsecondary 
Education Data System data collection; https://nces.ed.gov/programs/digest/d19/tables/dt19 105.50.asp 
© National Student Clearinghouse 2020. 
‘The NSC reports that, on average, 4 percent of students block the release of their records, but block rates vary across states and are highest among students 
enrolled in two-year colleges (National Student Clearinghouse Research Center 2017). 


® Though the wrong student could be linked with a given record in the NSC data, researchers have found that the NSC matching algorithm tends to result in more 
errors where no link is made than where the wrong student is linked. That is, research suggests students are more likely to not be found than to be mistaken for 


Student records are not 
suppressed 


YES 
Primarily student SSN 


YES 

Specific date that aid was 
dispensed allows for 
adjustable enrollment periods 


another person (Dynarski, Hemelt, and Hyman 2015). 
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Student records are 
not suppressed 


NO 

All Upward Bound 
students are 
included 

NO 

18 months after 
expected high 
school graduation 


Because no single source maintains college enrollment records for all students, reliance on any one source is 
likely to underestimate college enrollment. To determine the most useful source or sources to measure college 
enrollment, the study collected and analyzed enrollment data for an earlier cohort of Upward Bound students 
who were expected to graduate high school in 2011. These historical enrollment data allowed the study to 
examine the common and distinct features of each data source and assess the feasibility of combining 
information from multiple sources. 


Each source—NSC, FSA, and Upward Bound APRs—produces a different initial college enrollment rate (defined as 
enrolled on October 1 in the fall after high school graduation) for the 25,992 Upward Bound students expected to 
graduate in 2011 (the “2011 cohort”) (Exhibit B.12). The NSC is the most common data source for measuring 
enrollment; using only its records yielded a college enrollment rate for the 2011 cohort of 62 percent. Using only 
the FSA data yielded an enrollment rate of 65 percent. Relying solely on the APR data resulted in a 78 percent 
enrollment rate. Combining data from the NSC and the FSA yielded a college enrollment rate of 75 percent, 
whereas combining data from all three sources yielded a college enrollment rate of 82 percent. 


Exhibit B.12 College Enrollment Rates by Data Source 


Students Enrolled 
Data Source (%) 
| NSC only 62.2 
FSA only 65.2 
APR only 78.4 
NSC and FSA 74.5 
NSC, FSA, and APR 81.8 | 


Notes: NSC and FSA college enrollment range is July 1, 2011, to June 30, 2012 (12 months). 
APR college enrollment range is June 1, 2011, to December 1, 2012 (18 months). 
Sample Sizes: 
NSC: N = 25,992, Missing = O. 
FSA: N = 25,992, Missing = 0. 
APR: N = 25,213, Missing = 779 students whose enrollment was not recorded in the APR data. 
Source: FSA 2016; NSC 2016; APR 2012. 


Exhibit B.13 below shows that, although the majority of students (51 percent) were reported as enrolled in all 
three datasets, some students were captured in only one or two of the datasets. Using any source alone would 
underestimate the college enrollment rate. 
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Exhibit B.13.| Overlap in College Enrollment across Data Sources 


Percentage of students 


100% 


80% 


1.4 ™ APR only 


@ FSA only 
60% 


@ NSC only 
2.2 

@ NSC and APR 
40% m FSA and APR 
m NSC and FSA 


BNSC, FSA, and APR 
20% 


0% 


Students enrolled in college 


Notes: NSC and FSA college enrollment range is July 1, 2011, to June 30, 2012 (12 months). 
APR college enrollment range is June 1, 2011, to December 1, 2012 (18 months). 

Sample Size: 25,992 students. 

Source: FSA 2016; NSC 2016; APR 2012. 


For this study, college enrollment data were first obtained from the NSC. However, there are some limitations in 
using only the NSC data to examine college enrollment that may result in an undercount of students enrolled: 


Missing colleges. Some colleges do not provide student enrollment data to the NSC. About half of the 
postsecondary institutions that accept federal financial aid funding participate in the NSC (just over 
3,600 of about 6,500 institutions). However, while the NSC does not cover all postsecondary 
institutions, it does cover institutions that enroll most college students. The institutions that participate 
in NSC enroll 97.8 percent of all students enrolled in postsecondary institutions.” 


Missing students. Even students enrolled at postsecondary institutions that participate in the NSC might 
not be included in the NSC, and these unreported enrollments are indistinguishable from non- 
enrollments. Colleges that provide records to NSC can opt to not share records for non-degree-seeking 
students, such as those in training programs.” In addition, under the Family Educational Rights and 
Privacy Act (FERPA), individual students can opt out of reporting their information for research 
purposes, ”' leading to incomplete coverage in student-level NSC data. Finally, some colleges do not 
provide NSC with the records of students who do not have a Social Security number, including 
undocumented and international students.” 


= NSC (2020) 


NSC encourages institutions to include non-degree-seeking students, but institutions could have different reporting policies. See 


www.studentclearinghouse.org/colleges/enrollment-reporting/faqs/. 


o NSC (2017) estimates that about 4 percent of students block their records, and approximately 4 percent of students in the study sample were blocked by the 
school or student. 
2 Precise records about the number of students affected by this policy are not available. 
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e Matching errors. NSC matches students’ college records based on name and birthdate,* and there 
could be a small amount of error in this matching process. ** 


If Find the Fit encouraged students to enroll in college or to enroll in different types of colleges, particularly those 
not participating in the NSC, then not having complete college enrollment data for some students could 
potentially bias the estimated effects of Find the Fit. For this reason, the study used FSA data and explored the 
possibility of using the Upward Bound APRs to increase the accuracy of the college enrollment measures. 


FSA data cover students who receive federal financial aid,?° which most Upward Bound students are expected to 
be eligible for because most are from low-income households.”° FSA data cover the same time period as the 
study’s enrollment outcomes. The study treated disbursement of financial aid funds on behalf of a student as an 
alternative indicator of college enrollment. Students who did not have a record of college enrollment from either 
the NSC or FSA were assumed to not be enrolled in college. 


The APR data are submitted by each Upward Bound project to the Department of Education and contained at 
least some data for every student who entered the UB projects. However, “college enrollment” in the APR is 
defined over an 18-month period (June 1, 2011, to December 1, 2012), spanning two academic years, and is coded 
as ‘yes’ or ‘no’ without any specified date range of enrollment. This time period does not align with this study’s 
enrollment outcomes, which were defined as “immediate enrollment” (as of October 1), “first-year enrollment” 
(July 1 to June 30), and so on. Further, UB projects are encouraged to use the NSC data for their APR reporting, 
so this data source might not contribute additional information beyond those data identified in the NSC. For 
these reasons and those discussed below, APR data were not used in this study’s measure of college enrollment. 


Additional Information about Measuring College Academic Undermatch 


The study defines college academic “undermatch” as a student not attending college or attending a college that is 
less selective than those at which the student would have a 90 percent probability of admission.’ Students not 
enrolled in college are included in the undermatch definition because all high school seniors could be admitted 
to at least a two-year college. 


The study’s calculation of undermatch consists of the following steps: 
1. Classify each college by selectivity level. 
2. Estimate a statistical (regression) model of acceptance to colleges at different selectivity levels. 


3. Predict the probability of acceptance into each selectivity level for each Upward Bound student in the 
study. 


4. Determine the highest selectivity level to which each student likely could be admitted—that is, for which 
the student is a good match. 


5. Identify the selectivity level of the college that each Upward Bound student attended. 


3 NSC uses a logic path to match student records and then verifies matches by weighted node: 
https://www.studentclearinghouse.org/colleges/studenttracker/faqs/. 

aa Using Michigan data, Dynarski, Hemelt, and Hyman (2015) found that the NSC matching algorithm did not add appreciable measurement error to 

identifying students. NSC reports having a match rate that is approximately 94 percent accurate, accounting for the 2.4 percent of student enrollment files 

for institutions that are not participating with NSC and approximately 3.6 percent matching errors. According to NSC, matching errors can be caused by a 

number of factors, including duplicate student records, missing Social Security numbers, and enrollment reporting submission errors. 

https://www.studentclearinghouse.org/colleges/studenttracker/faqs/. 

FSA provides financial aid to more than 11 million students each year (U.S. Department of Education, n.d.). Among first-time, full-time degree- or certificate- 

seeking students, 85 percent of students at four-year colleges and 78 percent of students at two-year colleges received financial aid in 2013-14 (Kena et al. 

2016). 

According to Upward Bound Annual Performance Reports, 87.6 percent of the students in the study were from low-income households. 

aE The report’s undermatch outcome includes all students except those who attended four-year colleges that (a) are specialty schools, such as art schools, as 
identified by Barron’s; (b) are not rated by Barron’s; or (c) could not be directly linked to Barron’s, such as four-year branch campuses that were not 
included in Barron’s. The study could not assess whether students who attended these types of four-year colleges were or were not undermatched. When 
assessing the selectivity level of the college the student attended, the study coded students who attended these types of four-year colleges as attending any 
four-year college but not as attending colleges of selectivity levels above that level. Thus, there was no missing data for the selectivity outcome. 
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6. Compare the selectivity of the college the student attended with the highest selectivity level to which the 
student was a “good match.” 


Details of the study’s approach used to calculate undermatch are as follows:*8 


Step 1: Classify each college by selectivity level 


The 2014 NCES-Barron’s Admissions Competitiveness Index was used to divide colleges into selectivity levels. 
Two adaptations were made to the Barron’s rankings: (1) its top two rankings (most competitive and highly 
competitive) were collapsed into one level, and (2) two-year and less-than-two-year was added as the lowest 
selectivity level (Barron’s ranks only four-year colleges).”° Four-year institutions that Barron’s did not rank, such 
as art schools, were excluded from the study’s undermatch analysis. 


These adaptations resulted in the following six selectivity levels: 
e Most or Highly Competitive 
e Very Competitive 
e Competitive 
e Somewhat Competitive 
e Other Four-Year College 


e Two-Year or Less-Than-Two-Year College 


Step 2: Estimate a statistical (regression) model of acceptance to colleges at different selectivity levels 


Next, a statistical model was estimated using a nationally representative sample of 14,015 high school seniors 
(2003-04 seniors from the Education Longitudinal Study of 2002, or ELS:2002) to predict students’ probability 
of acceptance to colleges of different selectivity levels based on their high school academic credentials.*° 


The model links the colleges that students applied to” with the six selectivity levels from Step 1 using institution 
codes from IPEDS. More specifically, these selectivity levels are matched to the colleges to which ELS students 
applied and the colleges to which they were accepted. For each of the six selectivity levels, a student then is 
considered to have applied to that level if he/she applied to one or more colleges with that selectivity rating, 
regardless of the number of colleges in that selectivity level to which the student applied. Likewise, a student is 
considered to have been accepted to a selectivity level if the student applied and was accepted to one or more 
colleges with that selectivity rating, regardless of the number of colleges in that selectivity level to which the 
student was accepted. 


For each selectivity level, a binary variable was created indicating whether a student was accepted to at least one 
college in that level. For example, the binary variable for the very competitive level was coded as 1 if a student was 
accepted to one or more very competitive colleges and O if not. The coding of one selectivity level was not 
affected by acceptance to other selectivity levels. 


a This approach is similar to that used by Smith, Pender, and Howell (2013). However, unlike their analysis, this study’s analysis does not collapse the 

somewhat competitive and noncompetitive (or “other four-year”) levels of four-year colleges because Upward Bound students are more likely to attend less 

selective colleges than are U.S. high school students overall (authors’ comparison of data from Seftor, Mamun, and Schirm [2009] on Upward Bound 

students’ college enrollment versus data from Schmitt [2015] on national patterns of college enrollment). Additionally, students’ unweighted GPA was used 

instead of their weighted GPA because the weighted GPA was not available for students in the study sample. 

Two-year or less-than-two-year was included as one of the selectivity levels because approximately 25 percent of Upward Bound students in a prior study 

attended such colleges (Seftor, Mamun, and Schirm 2009). 

30 The ELS sample used in this analysis is restricted to students who were seniors in spring 2004, who applied to colleges, and for whom acceptance decisions 
are known. 

cs The ELS collected data on the number of colleges to which a student applied, the name of each college, and the applicant’s acceptance status at each. 
Students applied to between zero and 18 colleges. About 17 percent of students did not apply to any college, 51 percent applied to one or two colleges, and 
32 percent applied to three or more colleges. 
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The model predicting college acceptance included only measures of students’ academic credentials that were 
available in both the ELS data and the Upward Bound APR data, plus NSC, College Board, and ACT records. 
These characteristics were unweighted high school GPA; highest college entrance exam score (the SAT; the ACT 
converted to the SAT scale; or, if both the SAT and ACT were unavailable, the PSAT or PLAN), and whether the 
student had taken one or more Advanced Placement (AP) exams.” Students who were missing college entrance 
exam scores (32 percent of the sample, n = 4,497) were included using the dummy variable method (see Section 
B.4.2 for details on this method). An indicator for whether the SAT, ACT, or PSAT/PLAN score was used also was 
included in the model. Students in the ELS who were missing GPA (7 percent of the sample, n = 963) were 
excluded from the analysis because of the centrality of valid GPAs to the predictions. ** 


Analysis Model. Logistic regression was used to model the probability of acceptance to each of the six selectivity 
levels, conditional on the student having applied to a college at that selectivity level.*4 For example, only 
students who applied to competitive colleges were included in the analysis of the probability of acceptance to 
competitive colleges. 


The model of the probability of acceptance for the K" college selectivity level is conditional on application to that 
selectivity level, GPA, college entrance exam score, and AP test taking. It can be represented as: 


Prob[ACCEPTx = 1 | APPLY = 1, GPA, EXAM SCORE, AP TESTTAKING] 


More specifically, the following equation is estimated for each of the K = 6 college selectivity levels. The equation 
includes higher-order terms to allow the effects of GPA and college entrance exam score to vary at different 
points on the distribution—for example, enabling the effect of GPA to be more pronounced for higher GPAs than 
for moderate GPAs. The model also includes an interaction between GPA and missing exam score to allow the 
effect of GPA to differ for students with a non-missing exam score versus students with a missing exam score.*° 


1 


—(Bo+B1 GPA+P2GPA-SQ+3GPA-cubic+B,Hi-SAT—ACT—PSAT+f5EXAM-—SQ+f,6TEST-cubic 
+B7ACT-SCORE-—USED+fBgPSAT—SCORE-USED+ Bog Missing—Score—Flag+$4)APEXAM 
1l+e +B41,GPA*Missing—Score) 


Prob[ACCEPT, = 1] = 


Where, 


By is the covariate-adjusted log-odds of acceptance to a college of a given selectivity level for students 
who applied to a college in selectivity level K. 


ACCEPT, is a binary variable indicating acceptance into one or more colleges in selectivity level K. 
APPLY« is a binary variable indicating application to one or more colleges in selectivity level K. 
GPA is the student’s unweighted high school GPA. 

GPA-SQ is the student’s unweighted high school GPA squared (GPA*GPA). 

GPA-cubic is the student’s unweighted high school GPA cubed (GPA*GPA*GPA). 


Hi_SAT_ACT PSAT is the higher of the student’s most recent SAT or ACT score converted to the SAT 
scale, or is the student’s most recent PSAT score converted to the SAT scale if both the SAT and ACT 
score are unavailable, with missing values set to = O. 


ie There were a few differences between how academic credentials were measured in the ELS data versus the Upward Bound data. If Upward Bound students 

did not have a SAT, ACT, or PSAT score, their PLAN score was used instead, if available. PLAN scores were not available in the ELS data. Similarly, the 

variable for Upward Bound was whether students took one or more AP or International Baccalaureate courses. In the ELS data, whether students took one 

or more AP exams was used. 

For GPA, the dummy variable method to handling missing data was considered, but the approach resulted in estimation problems. 

a Modeling the probability of acceptance conditional on application follows Smith, Pender, and Howell’s (2013) approach, although the study team 
acknowledges that there might be unobserved characteristics that affect both application and acceptance. 

= The final model used to predict the probability of acceptance for each college selectivity category includes higher-order terms for GPA and highest test 
score (SAT, ACT, or PSAT) and interactions. Based on a test of goodness-of-fit (differences in -2 log likelihood), we determined that the model with cubic 
terms for GPA and test score and with the interaction between GPA and missing test score provides superior fit compared to models without these terms. 
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TEST-SQ is the student’s exam score squared. 
TEST-cubic is the student’s exam score cubed. 


ACT_ SCORE USED is a binary variable indicating that either the student took only the ACT and not the 
SAT or the student took both exams and scored higher on the ACT than the SAT. 


PSAT_ SCORE USED is a binary variable indicating the student took only the PSAT and not the ACT or 
SAT. 


MISSIN _SCORE_FLA is a binary variable indicating that the student’s SAT, ACT, and PSAT scores are all 
missing. 


APTEST is a binary variable indicating that the student took one or more AP tests. 


GPA*Missing-Score is the product of GPA and the binary indicator for students with missing SAT, ACT, 
and PSAT scores. Values are O for students with an exam score, and values are GPA for students without 
an exam score. 


The analysis is adjusted for the two-level, clustered sampling design (students clustered within schools) and non- 
response, using sampling and non-response weights. 


Step 3: Predict the probability of acceptance into each selectivity level for each Upward Bound student in the 
study 


Next, the study used the model described above to predict the probability of Upward Bound students’ 
acceptance to colleges at different selectivity levels. Using the parameter estimates generated from the ELS 
sample (in Step 2) for each selectivity level and each Upward Bound student’s academic credentials, the study 
estimated the probability of a student being accepted to colleges in each of the selectivity levels. 


Step 4: Determine the highest selectivity level to which each student likely could be admitted—that is, for 
which the student is a good match 


Next, the study defined the highest selectivity level to which each student likely would have access, following the 
approach used by Smith and his coauthors.*° This approach defines the highest selectivity level to which a 
student has at least a 90 percent probability of acceptance, conditional on applying (to that selectivity level).*’ 


In rare cases, students could have a greater than 90 percent conditional probability of being accepted to a given 
selectivity level but a less than 90 percent chance of being accepted to a lower selectivity level(s). In such cases, 
students were classified as having access to the lowest level that was less than 90 percent. *® 


Step 5: Identify the selectivity level of the college that each Upward Bound student attended 


To identify the colleges that students attended in fall 2016, the study linked Upward Bound students to data on 
college attendance from the NSC and FSA. The study defined students as attending college if either the NSC or 
FSA data recorded them as enrolled on October 1, 2016. If a student was enrolled in multiple colleges on October 
1, full-time enrollment superseded part-time enrollment. (More details on how the study measured college 
enrollment are provided in the prior section.) 


The NSC and FSA data include Office of Postsecondary Education college codes, which were then linked to the 
2015-16 IPEDS institution codes. The student’s college was then matched to the 2014 NCES-Barron’s data by the 
IPEDS institution code, to determine the college’s selectivity level. 


= Smith, Pender, and Howell (2013). 
This is also the threshold used by Roderick et al. (2008) to define the highest selectivity level to which students have access. 
This is consistent with the approach taken by Smith, Pender, and Howell (2013). 
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Step 6: Compare the selectivity of the college the student attended with the highest selectivity level to which 
the student was a “good match” 


For each student, the study compared the selectivity of the college where the student enrolled versus the highest 
selectivity level to which the student was predicted to have at least a 90 percent conditional probability of 
acceptance. If the student enrolled in a college at or above the highest predicted selectivity level of access, the 
student was classified as matched. If the student enrolled at a lower level, or did not enroll in college at all, the 
student was classified as undermatched. 


Measures Describing Students and Projects 


The study uses characteristics measured before the lottery (at baseline) to describe Upward Bound projects and 
students. Data for most student characteristics come from the 2015 APRs, which as described previously, were 
submitted by each Upward Bound project to the Department of Education and contained data for almost every 
student who entered the projects.*° Data were available for more than 98 percent of the study sample in the 
APR. When missing from the APR, data on student gender, race/ethnicity, and “first generation to college” status 
were taken from the survey that students completed prior to the lottery. For students’ college entrance exam 
scores, students’ highest score on the SAT or ACT through spring 2015 was used (or their PSAT or PLAN score*° 
was substituted if SAT and ACT scores were not available). 


Fortunately, information about student characteristics from the APR and the baseline student survey were very 
well aligned, making the survey a good alternative source for filling in missing data as needed. Exhibit B.14 shows 
the congruence of data from the two sources when data were available in both sources. For measures available 
in both the APR and baseline student survey, the data matched for 89 to 98 percent of students. 


Exhibit B.14 Congruence between Annual Performance Report and Baseline Student Survey Data 


Percentage of Students Missing Data Number of 
Baseline Students in Percentage 
| Measure APR Survey Both Both Sources _—_ Congruent 
Gender 1.0 23.1 0.5 3,395 98.3 
Race/ethnicity Dep 23.8 0.7 3,320 93.7 
First generation to college 1.0 33.6 0.6 2,932 88.5 


Notes: Only students with non-missing values in both the APR and baseline student survey are used to calculate the percentage congruent between the two data 
sources. 

Sample Size: For percentage missing = 4,443 students. 

Source: APR 2014-15; baseline student survey 2015. 


= The APR data was the preferred source of data for most of the baseline student characteristics rather than the baseline student survey because the APR had 
more complete data for the study sample. Data were missing for at least 20 percent of the sample in the baseline student survey data, compared to less than 
2 percent missing data in the APR. 

40 Students’ college entrance exam scores were collected when students in the study sample were in the 10" or 11" grade; as a result, some students had only 
PSAT or PLAN scores available. These tests are strong predictors of performance on the SAT and can be converted to a predicted SAT score using the 
conversion tables provided by the College Board. 
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Exhibit B.15 below shows the definition of each student characteristic, its data source, and the percentage of 
students missing data. In addition, the exhibit shows the same information for the Upward Bound project 
characteristics—host institution type, locale, and project historical college enrollment rate. 


Some of the student characteristics listed in Exhibit B.15 are also used as covariates in the statistical models 
examining the effects of Find the Fit to take into account possible existing differences between students in the 
Find the Fit and regular UB advising group projects and to improve the precision of the impact estimates. *! The 
analytic models also account for host institution type and locale via the randomization blocks discussed in 
Section B.4 and include projects’ historical college enrollment rate as an additional project-level baseline 
measure. 


- Each analysis model included one of the two following student characteristics: (a) whether the student planned to apply to college as of the spring of her/his 
junior year or (b) whether the student would be undermatched at the highest selectivity college to which she/he planned to apply as of the spring of her/his 
junior year. Specifically, whether the student would be undermatched based on the colleges she/he planned to apply to at baseline was included as a 
covariate in the models estimating impacts on undermatch and the alternative measures of college quality, while whether the student was planning to apply 
to college was included as a covariate in the remaining models. 


36 


Exhibit B.15 Student and Project Characteristics Before the Lottery: Construction and Missing Data 


Student Characteristic 
Gender 


Race/ethnicity 


Low-income 
household 


First generation to 
college 


Completed AP/IB 
course 

GPA 

College entrance exam 
score 

Planned to apply to 
college 


Undermatched at all 
colleges where 
planned to apply 


APR; baseline 
student survey 


APR; baseline 
student survey 


APR; baseline 
student survey 


APR 


APR 


College Board 
(SAT) or ACT 


Baseline student 
survey 


Baseline student 
survey 


1= Female 
O = Male 


Four categories: 

1 = Hispanic 

2 = White, non-Hispanic 

3 = Black, non-Hispanic 

4 = Other/multiracial, non-Hispanic 
1= Upward Bound eligibility criteria indicate that household is low 
income 
O = Upward Bound eligibility criteria do not indicate that 
household is low income 
1 = No parent in the household received a bachelor’s degree 
O = At least one parent in the household received a bachelor’s 
degree 
1 = Completed an AP or IB course or both 
O = Has not completed an AP or IB course 
Unweighted grade point average 
Highest score on SAT or ACT, or PSAT or PLAN if no SAT or ACT 
score available; all scores converted to SAT scale 
1 = Student reported planning to apply to college 
O = Student did not report planning to apply to college 
1 = Student would be undermatched at the highest selectivity 
college where planning to apply, or student did not report 
planning to apply to college 
O = Student would not be undermatched at the highest selectivity 
college where planning to apply 
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1.1 


11 


0.8 


1.3 


15.7 
25.3 


19.4 


21.2 


0.4 


0.9 


0.3 


1.0 


12.0 
28.9 


19.3 


20.8 


0.7 


1.0 


0.6 


1.2 


13.9 
27.0 


19.4 


21.0 


Project Characteristic 


Host institution type IPEDS Three categories: 
3 = Four-year college 
2 = Two-year college 
1 = Other 
Locale IPEDS Four categories: 
1= City 
2 = Suburb 
3 = Town 
4 = Rural 


Project historical APR Project’s reported rate of college enrollment for 2012-13 high 
college enrollment rate school graduates 


0.0 


0.0 


5.1 


0.0 0.0 


0.0 0.0 


3.1 41 


AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 
Sample Sizes: 
Student characteristics: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 
Project characteristics: Find the Fit group = 98 projects, Regular UB advising group = 96 projects. 
Source: APR 2012-13 and 2014-15; baseline student survey 2015; college entrance exam score data 2015; IPEDS 2015-16. 
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Several of the baseline characteristics described above were also used in exploratory analyses to examine the 
effectiveness of Find the Fit for subgroups of students or projects. The study included five subgroups that were of 
policy interest and for which there was existing research suggesting that enhanced advising such as Find the Fit 
might have different impacts. For example, prior research suggests that some types of students are likely to have 
higher rates of college undermatch or face greater challenges in attending more selective, higher-quality 
institutions. Exposure to Find the Fit could affect such students differently than students with fewer challenges. 
Understanding impacts on these particular subgroups could be useful for targeting technical assistance or other 
aspects of program improvement. Exhibit B.16 shows these subgroups and the rationale for examining each 


group. 


Exhibit B.16 Rationale for Exploring Effects for Subgroups of Students and Projects 


Number (Percentage) of Study 
Students by Subgroup Category 


Subgroup Rationale 


Student Characteristic 


Gender Male students are more likely to undermatch than female Male: 1,592 (36.0%) 
students? and their college enrollment rates are lower.’ Thus, Female: 2,828 (64.0%) 
identifying strategies that improve male students’ enrollment 
is of particular policy interest. 


Race/ethnicity Students’ race/ethnicity is related to their probability of Hispanic: 1,139 (25.8%) 
undermatch. For example, controlling for other factors, Black White, non-Hispanic: 1,031 (23.4%) 
students are less likely to undermatch than are other Black, non-Hispanic: 1,704 (38.6%) 


students.‘ Hispanic students are particularly likely to attend Other: 536 (12.2%) 
local colleges,‘ potentially creating a barrier to reducing 


undermatch. 
College Many colleges use SAT and ACT exam scores as a factor for Highest quartile: 218 (4.9%) 
entrance admission and a proxy for academic preparation in high Second quartile: 533 (12.0%) 
exam score® school. Given that some approaches to address undermatch Third quartile: 826 (18.6%) 
have focused exclusively on students with high academic Lowest quartile: 1,667 (37.5%) 
qualifications,‘ one purpose of this study is to understand Missing score: 1,199 (27.0%) 


whether undermatch can be ameliorated for students with 
varying levels of academic qualifications. 


JM) (cam OteacouCite 


Rural host Students in rural areas may have fewer colleges to choose Rural: 461 (10.4%) 
institution from nearby, which may partly explain why students from Non-rural: 3,982 (89.6%) 

rural areas are more likely to undermatch in their college 

choices.‘ 
Host The study’s analysis of historical APR data found that students Four-year college: 2,819 (63.4%) 
institution at Upward Bound projects hosted by two-year colleges enroll Two-year college: 1,069 (24.1%) 
type in their host institution at a higher rate than do students at Other: 555 (12.5%) 


projects hosted by four-year colleges. 


a Smith, Pender, and Howell (2013). The rate of undermatch for male students was 3 percentage points higher than for female students. 

b Bailey and Dynarski (2011). 

c Roderick, Coca, and Nagaoka (2011); Smith, Pender, and Howell (2013). The rate of undermatch for Black students was 9 percentage points lower than for white 
students and the rate of undermatch for Hispanic students was 5 percentage points lower than for white students 

d Hurtado et al. (2008). 

e For subgroup analyses based on students’ college entrance exam scores, students’ scores were coded into four categories based on quartiles from the nationally 
representative ELS:2002, providing a point of reference for these scores beyond the study sample. The quartiles were 1140-1600, 990-1130, 860-980, and 400- 
850. 

f For example, Hoxby and Turner (2013). 

g Smith, Pender, and Howell (2013). 
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B.4.2 Estimating the Effectiveness of Find the Fit 


The study defined the “sample” as all participating Upward Bound projects and their rising 2015-16 seniors who 
participated in the lottery. This definition helped to ensure that the estimates reflected unbiased effects of Find 
the Fit on students. Exhibit B.17 below diagrams the flow of study participants from the recruited Upward Bound 
projects and students to the analytic study samples used to investigate the effectiveness of Find the Fit on each of 
the measures in this report. 


Exhibit B.17_ Flow from Recruitment of Upward Bound Projects to Study Samples 
Sample Used for Analysis of the Effectiveness of Find the Fit 


Recruited projects from FY 2012 grant cycle with rising high school seniors in 2015-16 (n=702) 
Volunteered for the study: 


(Project n=194) 
(Student n=4,454) 


ne P Excluded students lacking 


Random assignment parental consent 
(Student n=11) 


Assigned to integrate Find the Fit into regular Upward Bound 
services - Find the Fit group 
(Project n=98) 
(Student n=2,336) 


Obtained outcome data from: 


Assigned to offer regular Upward Bound services only - 
Regular UB advising group 
(Project n=96) 
(Student n=2,107) 


Obtained outcome data from: 


IPEDS" Federal Student Aid office IPEDS Federal Student Aid office 
(Project n=98) (Project n=98) (Project n=96) (Project n=96 

(Student n=1,516) (Student n=2,336) (Student n=1,683) (Student n=2,107) 
National Student Clearinghouse Annual Performance Report National Student Clearinghouse Annual Performance Report 
(Project n=98) (Project n=98) (Project n=96) (Project n=96) 

(Student n=2,336) (Student n= 2,095) (Student n=2,107) (Student n=1,942) 

College entrance exam scores College entrance exam scores 

(Project n=98) (Project n=96) 

(Student n=1,745) (Student n=1,499) 

NCES-Barron’s Admissions Competitiveness Index NCES-Barron’s Admissions Competitiveness Index 

(Project n=98) (Project n=96) 

(Student n=2,335) (Student n=2,107) 


Created analytic samples: 


Entire randomized sample 
Persistence* 

(Project n=194) 

(Student n=4,443) 


Undermatch Transfer to less selective Attend familiar 
(Project n=194) 


(Student n=4,270) 


College selectivity* 
(Project n=194) 
(Student n=4,443) 


college or dropout college 


(host institution) 
(Project n=194) 
(Student n=4,443) 


(Project n=194) 
(Student n=4,443) 


Subsample of randomized students - those enrolled in college 
Exploratory analyses of characteristics of colleges attended in year after high school 


Distance from home 
(Project n=194) 
(Student n=2,759) 


College entrance exam College graduation 
scores of incoming rates 
freshmen (Project n=194) 


(Project n=194) (Student n=2,755) 
(Student n=1,811) 


Sticker price Net price 
(Project n=194) (Project n=194) 
(Student n=2,749) (Student n=2,755) 


*Notes: Sample sizes are the same for each year (first, second, and third years after high school). 
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Random assignment by lottery created two groups—Find the Fit, regular UB advising—that were initially similar 
on characteristics related to college academic undermatch, selectivity, and persistence (Exhibits B.5 and B.6, 
above). Together, the two groups comprise the full randomized sample. However, some analyses focused on 
smaller groups of students, such as those who initially enrolled in college or those for whom academic 
undermatch could be calculated. For each analysis that did not use the full sample, the baseline characteristics of 
students in the two groups are compared to establish that the groups were similar before the lottery. Thus, any 
differences seen in college-going behaviors between the two groups could be attributed to Find the Fit. 


College Academic Undermatch. The students in the Find the Fit and regular UB advising groups in the sample 


used to analyze the effectiveness of Find the Fit on college academic undermatch had similar characteristics 
before the lottery (Exhibit B.18). 


Exhibit B.18 Characteristics of Students before the Lottery Used in the Analysis of Undermatch, 
by Group 


Female 64.4 64.5 -0.2 921 -0.004 
Race/Ethnicity 

Hispanic 26.9 21.9 5.0 261 0.163 
White, non-Hispanic 22.4 25.5 -3.0 .455 -0.101 
Black, non-Hispanic 38.8 39.6 -0.9 .872 -0.022 
Other, non-Hispanic 11.9 13.0 -1.1 71 -0.061 
Low-income household 88.1 87.7 0.4 .782 0.021 
First generation to college 91.1 92.4 =1:3 .266 -0.102 
Taken one or more AP/IB courses 36.4 31.5 4.8 194 0.131 
Unweighted cumulative GPA (mean) 3.1 3.0 0.0 576 0.039 
College entrance exam (mean SAT score) 874.6 864.3 10.3 396 0.061 
Undermatched at all colleges where 19.5 21.6 2.2 273 -0.080 
planned to apply 

Project historical college enrollment rate 84.0 86.2 -2.2 273 -0.200 
F-test® p= .657 


AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 

a p-Values shown in this column are for tests of whether there was a statistically significant baseline difference in the characteristic indicated in the row. 

b Effect sizes are calculated using the Hedges’ G formula for continuous variables and the Cox Index for binary variables. 

c p-Value shown in this row is for a test of whether the Find the Fit and regular UB advising groups statistically differed overall at baseline, across all 
characteristics shown in the table. 

Notes: The Find the Fit group percentage and estimated difference are adjusted for the blocked random assignment design and the clustering of students within 
Upward Bound projects. See Exhibit B.15 for additional details on missing data for characteristics measured before the lottery. The unadjusted standard 


deviations for the Find the Fit and regular UB advising groups, respectively, were 0.6 and 0.7 for cumulative GPA and 164.2 and 172.0 for college entrance exam. 
Sample Sizes: 


Gender: Find the Fit group = 2,256 students, Regular UB advising group = 1,991 students. 

Race/ethnicity: Find the Fit group = 2,249 students, Regular UB advising group = 1,988 students. 
Low-income household: Find the Fit group = 2,248 students, Regular UB advising group = 1,978 students. 
First generation to college: Find the Fit group = 2,255 students, Regular UB advising group = 1,990 students. 
Taken AP/IB courses: Find the Fit group = 2,243 students, Regular UB advising group = 1,975 students. 

GPA: Find the Fit group = 1,916 students, Regular UB advising group = 1,758 students. 

College entrance exam: Find the Fit group = 1,710 students, Regular UB advising group = 1,423 students. 
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Undermatched at all colleges where planned to apply: Find the Fit = 1,793 students, Regular UB advising group = 1,586 students. 
Project historical college enrollment rate: Find the Fit group = 2,257 students, Regular UB advising group = 1,962 students. 
Source: APR 2012-13 to 2014-15; college entrance exam score data 2015; baseline student survey 2015. 


College Entrance Exam Scores. The students in the Find the Fit and regular UB advising groups in the sample 
used to analyze the academic qualifications of incoming freshmen at the colleges attended had similar 
characteristics before the lottery (Exhibit B-19). The sample used for this analysis is smaller than the samples 
used for other analyses because most two-year colleges and some four-year colleges, particularly specialty 
schools and noncompetitive colleges, do not require students to submit standardized test scores. 


Exhibit B.19 Characteristics of Students before the Lottery Used in the Analysis of College Entrance 
Exam Scores of Incoming Freshmen at College Attended, by Group 


Female 67.3 68.1 -0.8 .747 -0.023 
Race/Ethnicity 

Hispanic 29.6 22.2 7.4 134 0.235 

White, non-Hispanic 17.6 22.5 -4.9 .267 -0.185 
Black, non-Hispanic 39.4 40.3 -0.9 .877 -0.023 
Other, non-Hispanic 13.3 14.9 -1.7 -610 -0.083 
Low-income household 87.0 86.1 0.9 .673 0.046 
First generation to college 91.8 90.9 0.9 .583 0.072 

Taken one or more AP/IB courses 51.6 47.1 4.5 346 0.110 

Unweighted cumulative GPA (mean) 3.3 3.3 -0.0 .448 -0.062 
College entrance exam (mean SAT score) 927.2 924.3 2.9 852 0.017 

Buck pias Seger ial 15.0 17.1 -2.1 377 0.094 
Project historical college enrollment rate 84.8 86.9 -2.1 377 -0.214 
F- test® p=.5ll 


AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 
@ p-Values shown in this column are for tests of whether there was a statistically significant baseline difference in the characteristic indicated in the row. 
Effect sizes are calculated using the Hedges’ G formula for continuous variables and the Cox Index for binary variables. 
© p-Value shown in this row is for a test of whether the Find the Fit and Regular UB advising groups statistically differed overall at baseline, across all 
characteristics shown in the table. 
Notes: Find the Fit group percentage and estimated difference are adjusted for the blocked random assignment design and the clustering of students within 
Upward Bound projects. See Exhibit B.15 for additional details on missing baseline data. The unadjusted standard deviations for the Find the Fit and Regular UB 
advising groups, respectively, were 0.6 and 0.7 for cumulative GPA and 163.7 and 171.8 for college entrance exam. 
Sample Sizes: 

Gender: Find the Fit group = 986 students, Regular UB advising group = 825 students. 

Race/ethnicity: Find the Fit group = 982 students, Regular UB advising group = 823 students. 

Low-income household: Find the Fit group = 986 students, Regular UB advising group = 825 students. 

First generation to college: Find the Fit group = 986 students, Regular UB advising group = 825 students. 

Taken AP/IB course: Find the Fit group = 985 students, Regular UB advising group = 822 students. 

GPA: Find the Fit group = 986 students, Regular UB advising group = 825 students. 

College entrance exam score: Find the Fit group = 795 students, Regular UB advising group = 618 students. 

Planned to apply to college: Find the Fit group = 827 students, Regular UB advising group = 692 students. 

Project historical college enrollment rate: Find the Fit group = 981 students, Regular UB advising group = 817students. 
Source: APR 2012-13 to 2014-15; college entrance exam score data 2015; baseline student survey 2015. 
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Graduation Rate. Similarly, the students in the Find the Fit and regular UB advising groups in the sample used to 
analyze the graduation rates of the colleges attended had similar characteristics before the lottery (Exhibit 20). 


Exhibit B.20 Characteristics of Students before the Lottery Used in the Analysis of Graduation Rate at 
College Attended, by Group 


Female 65.5 66.2 -0.7 .739 -0.018 
Race/Ethnicity 

Hispanic 28.8 24.1 4.7 310 0.147 
White, non-Hispanic 20.2 24.7 -4.5 285 -0.157 
Black, non-Hispanic 38.6 38.3 0.4 .950 0.009 
Other, non-Hispanic 12.2 12.9 -0.7 813 -0.039 
Low-income household 88.0 86.3 1.8 .287 0.095 
First generation to college 91.8 91.9 -0.1 942 -0.008 
Taken one or more AP/IB courses 42.8 38.2 4.6 .279 0.115 
Unweighted cumulative GPA (mean) 3.2 3.2 0.0 .899 0.009 
College entrance exam (mean SAT score) 896.6 886.8 9.8 478 0.057 
Pee Sa a 16.8 18.8 -2.0 351 -0.084 
Project historical college enrollment rate 84.2 86.3 -2.0 351 -0.201 
F- test* p=.607 


AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 
# p-Values shown in this column are for tests of whether there was a statistically significant baseline difference in the characteristic indicated in the row. 
Effect sizes are calculated using the Hedges’ G formula for continuous variables and the Cox Index for binary variables. 
© p-Value shown in this row is for a test of whether the Find the Fit and Regular UB advising groups statistically differed overall at baseline, across all 
characteristics shown in the table. 
Notes: Find the Fit group percentage and estimated difference are adjusted for the blocked random assignment design and the clustering of students within 
Upward Bound projects. See Exhibit B.15 for additional details on missing baseline data. The unadjusted standard deviations for the Find the Fit and Regular UB 
advising groups, respectively, were 0.6 and 0.6 for cumulative GPA and 164.3 and 177.9 for college entrance exam. 
Sample Sizes: 

Gender: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

Race/ethnicity: Find the Fit group = 1,418 students, Regular UB advising group = 1,330 students. 

Low-income household: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

First generation to college: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

Taken AP/IB course: Find the Fit group = 1,421 students, Regular UB advising group = 1,330 students. 

GPA: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

College entrance exam score: Find the Fit group = 1,098 students, Regular UB advising group = 976 students. 

Planned to apply to college: Find the Fit group = 1,177 students, Regular UB advising group = 1,096 students. 

Project historical college enrollment rate: Find the Fit group = 1,414 students, Regular UB advising group = 1,310 students. 
Source: APR 2012-13 to 2014-15; college entrance exam score data 2015; baseline student survey 2015. 
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Sticker Price. In the sample used to analyze the sticker prices of the colleges that students attended, students in 
the Find the Fit and regular UB advising groups had similar characteristics before the lottery, except that students 
in Find the Fit projects were 2 percentage points more likely than their counterparts in regular UB advising 
projects to report that they planned to apply to college (Exhibit B.21). This difference could be expected by 
chance, given the number of characteristics examined. Even so, the difference was taken into account in the 
statistical model used to estimate the effect of Find the Fit. The difference (effect size = 0.20) is within the range 
the What Works Clearinghouse™ considers acceptable if the analysis models control for the difference.” 


Exhibit B.21. Characteristics of Students before the Lottery Used in the Analysis of Sticker Price at 


College Attended, by Group 


Female 752 -0.018 
Hispanic 28.9 24.2 .309 0.147 

White, non-Hispanic 20.2 24.8 ry .282 -0.158 

Black, non-Hispanic 38.6 38.2 0.4 .946 0.010 
Other, non-Hispanic 12.2 12.9 -0.7 812 -0.039 
Low-income household 88.0 86.3 316 0.089 
First generation to college 91.8 91.9 m5 ; 927 -0.010 
Taken one or more AP/IB courses 42.8 38.3 281 0.114 

Unweighted cumulative GPA (mean) 3.2 3.2 . .899 0.009 
Coles entrance exam (mean SAT score) 896.6 886.7 A471 0.058 
Planned to apply to college 92.9 90.5 .043 0.195 

Project historical college enrollment rate 84.2 a6. 3 z 0 355 -0.200 
F-test* p= .622 


* p-Values shown in this column are for tests of whether there was a statistically significant baseline difference in the characteristic indicated in the row. 
> Effect sizes are calculated using the Hedges’ G formula for continuous variables and the Cox Index for binary variables. * p-Value shown in this row is for a test of 
whether the Find the Fit and Regular UB advising groups statistically differed overall at baseline, across all characteristics shown in the table. 
AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 
Notes: Find the Fit group percentage and estimated difference are adjusted for the blocked random assignment design and the clustering of students within 
Upward Bound projects See Exhibit B.15 for additional details on missing baseline data. The unadjusted standard deviations for the Find the Fit and Regular UB 
advising groups, respectively, were 0.6 and 0.6 for cumulative GPA and 164.4 and 178.0 for college entrance exam. 
Sample Sizes: 

Gender: Find the Fit group = 1,418 students, Regular UB advising group = 1,331 students. 

Race/ethnicity: Find the Fit group = 1,414 students, Regular UB advising group = 1,328 students. 

Low-income household: Find the Fit group = 1,418 students, Regular UB advising group = 1,331 students. 

First generation to college: Find the Fit group = 1,418 students, Regular UB advising group = 1,331 students. 

Taken AP/IB course: Find the Fit group = 1,417 students, Regular UB advising group = 1,328 students. 

GPA: Find the Fit group = 1,418 students, Regular UB advising group = 1,331 students. 

College entrance exam score: Find the Fit group = 1,095 students, Regular UB advising group = 975 students. 

Planned to apply to college: Find the Fit group = 1,201 students, Regular UB advising group = 1,116 students. 

Project historical college enrollment rate: Find the Fit t group = 1,410 students, Regular UB advising group = 1,308 students. 
Source: APR 2012-13 to 2014-15; college entrance exam score data 2015; baseline student survey 2015. 


Net Price. In the sample used to analyze the net price of the colleges that students attended, students in the Find 
the Fit and regular UB advising groups had similar characteristics before the lottery, except that students in Find 


2 The What Works Clearinghouse™ considers this an acceptable way to establish baseline equivalence on characteristics with a mean difference between the 


treatment and control group that is less than .25 standard deviations, which is the case for all baseline characteristics measured in this study. 


44 


the Fit projects were 2 percentage points more likely than students in regular UB advising projects to report that 
they planned to apply to college (Exhibit B.22). Again, this difference (effect size = 0.20) could be expected by 
chance, given the number of characteristics examined, and the difference was taken into account in the 
statistical model used to estimate the effect of Find the Fit. 


Exhibit B.22 Characteristics of Students before the Lottery Used in the Analysis of Net Price at College 
Attended, by Group 


Female 65.5 66.2 -0.7 .739 -0.018 
Race/Ethnicity 

Hispanic 28.8 24.1 4.7 310 0.147 
White, non-Hispanic 20.2 24.7 -4.5 285 -0.157 
Black, non-Hispanic 38.6 38.3 0.4 .950 0.009 
Other, non-Hispanic 12.2 12.9 -0.7 813 -0.039 
Low-income household 88.0 86.3 1.8 287 0.095 
First generation to college 91.8 91.9 -0.1 942 -0.008 
ans Cee ae 42.8 38.2 4.6 279 0.115 
Unweighted cumulative 

ae me 3.2 3.2 0.0 899 0.009 
ec nree aiaee 896.6 886.8 9.8 478 0.057 
Planned to apply to college 92.9 90.5 2.4 .042 0.195 
aes aaa nee 84.2 86.3 -2.0 351 -0.201 
F-test? p= .607 


AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 
@ p-Values shown in this column are for tests of whether there was a statistically significant baseline difference in the characteristic indicated in the row. 
> Effect sizes are calculated using the Hedges’ G formula for continuous variables and the Cox Index for binary variables. * p-Value shown in this row is for a test of 
whether the Find the Fit and Regular UB advising groups statistically differed overall at baseline, across all characteristics shown in the table. 
Notes: Find the Fit group percentage and estimated difference are adjusted for the blocked random assignment design and the clustering of students within 
Upward Bound projects. See Exhibit B.15 for additional details on missing baseline data. The unadjusted standard deviations for the Find the Fit and Regular UB 
advising groups, respectively, were 0.6 and 0.6 for cumulative GPA and 164.3 and 177.9 for college entrance exam. 
Sample Sizes: 

Gender: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

Race/ethnicity: Find the Fit group = 1,418 students, Regular UB advising group = 1,330 students. 

Low-income household: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

First generation to college: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

Taken AP/IB course: Find the Fit group = 1,421 students, Regular UB advising group = 1,330 students. 

GPA: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

College entrance exam score: Find the Fit group = 1,098 students, Regular UB advising group = 976 students. 

Planned to apply to college: Find the Fit group = 1,203 students, Regular UB advising group = 1,117 students. 

Project historical college enrollment rate: Find the Fit group = 1,414 students, Regular UB advising group = 1,310 students. 
Source: APR 2012-13 to 2014-15; college entrance exam score data 2015; baseline student survey 2015. 
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Distance from Home. In the sample used to analyze the distance from home of the colleges that students 
attended, students in the Find the Fit and regular UB advising groups had similar characteristics before the 
lottery, except that students in Find the Fit projects were 2 percentage points more likely than students in regular 
UB advising projects to report at baseline that they planned to apply to college (Exhibit B.23). This difference 
(effect size = 0.19) is within the range the What Works Clearinghouse™ considers acceptable if the analysis 
models control for the difference, which the study’s models do. 


Exhibit B.23 Characteristics of Students before the Lottery Used in the Analysis of Distance from 
Home of College Attended, by Each Group 


Gender 

Female 65.4 66.2 -0.8 .693 -0.022 
Race/Ethnicity 

Hispanic 28.8 24.2 4.6 oon 0.143 
White, non-Hispanic 20.3 24.8 -4.5 .286 -0.156 
Black, non-Hispanic 38.6 38.2 0.4 .935 0.011 
Other, non-Hispanic 12.1 12.8 -0.7 814 -0.039 
Household Characteristic 

Low-income household 87.9 86.3 1.6 .318 0.089 
First generation to college 91.8 91.9 -0.1 935 -0.009 
Taken one or more AP/IB courses 42.9 38.2 4.7 .266 0.118 
Unweighted cumulative GPA (mean) 3.2 3.2 0.0 .887 0.010 
eR cone’ Csouumearent 896.9 886.7 10.2 457 0.060 
Planned to apply to college 92.9 90.5 2.4 .045 0.192 
Project historical college enrollment rate 84.3 86.2 -2.0 .370 -0.193 
F-test* p= .623 


AP is Advanced Placement. IB is International Baccalaureate. GPA is grade point average. 
@ p-Values shown in this column are for tests of whether there was a statistically significant baseline difference in the characteristic indicated in the row. 
> Effect sizes are calculated using the Hedges’ G formula for continuous variables and the Cox Index for binary variables. * p-Value shown in this row is for a test of 
whether the Find the Fit and Regular UB advising groups statistically differed overall at baseline, across all characteristics shown in the table. 
Notes: Find the Fit group percentage and estimated difference are adjusted for the blocked random assignment design and the clustering of students within 
Upward Bound projects. See Exhibit B.15 for additional details on missing baseline data. The unadjusted standard deviations for the Find the Fit and Regular UB 
advising groups, respectively, were 0.6 and 0.6 for cumulative GPA and 164.2 and 177.8 for college entrance exam. 
Sample Sizes: 

Gender: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 

Race/ethnicity: Find the Fit group = 1,419 students, Regular UB advising group = 1,333 students. 

Low-income household: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 

First generation to college: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 

Taken AP/IB course: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

GPA: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 

College entrance exam score: Find the Fit group = 1,099 students, Regular UB advising group = 979 students. 

Planned to apply to college: Find the Fit group = 1,204 students, Regular UB advising group = 1,120 students. 

Project historical college enrollment rate: Find the Fit group = 1,415 students, Regular UB advising group = 1,313 students. 
Source: APR 2012-13 to 2014-15; college entrance exam score data 2015; baseline student survey 2015. 
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Estimation Models 
This section describes the study’s approach to estimating the effects of Find the Fit on students’ college going. 


Main Estimation Model 


The overall effect of Find the Fit was estimated using a statistical model. Because students are clustered within 
Upward Bound projects, the effect of Find the Fit was analyzed using two-level hierarchical linear models with 
students (level-1) nested in projects (level-2).** Linear models with conventional standard errors were used 
instead of non-linear models, even for outcomes such as college academic undermatch that are binary. This is 
because linear models are simpler to estimate and to interpret, yield unbiased estimates of the treatment effect, 
yield standard error estimates that are approximately correct even when the underlying data-generating process 
is nonlinear, “* and have been used by many random assignment evaluations in education.* To estimate the 
effects of Find the Fit: 


The regression model’s level-1 (student-level) equation was: 
(1) ¥i; = Bo; + L4=1 Ba; (BaselineStudentCharacteristics,;) + &;; 


Where, 
i indexes students and j indexes projects. 
Y,; is the value of the outcome, such as undermatch, for the i” student in the j’" Upward Bound project. 
Bo; is the covariate-adjusted mean value“ of the outcome in project j. 
BaselineStudentCharacteristics;; is a set of student characteristics measured before the lottery.’ 
Bq; Tepresents the relationships between the baseline student characteristics and the outcome. 
€;; is random error, assumed to be identically, independently and normally distributed. 

The regression model’s level-2 (project-level) equation was: 


(2) Bo; = Yoo + Yo1Treatment, + Yo2Historical Enrollment; + ar Yon(RA_Blocks,;) + Uo; 


Where, 
Treatment; equals 1 for the Find the Fit group and O for the regular UB advising group. 
Historical Enrollment; equals the project’s historical college enrollment rate. 
RA_Blocks,; includes seven dummy variables representing the eight randomization blocks. 


The effect of Find the Fit is given by the parameter y,,. For binary variables, yo,is the estimated difference 
between the proportion of students in Find the Fit projects and the proportion of students in regular UB advising 
projects who had a value of 1 for the outcome variable—for example, students who were undermatched at the 


Students were grouped within Upward Bound projects in this study, hence there are sure to be commonalities or interdependence for students from the 
same project. A two-level hierarchical linear model was used to estimate impacts to ensure that the model correctly accounts for the correlation among 
students within a project. 

a Judkins and Porter (2015). 

#5 Examples include the evaluation of the Teacher Incentive Fund (Max et al. 2014) and the evaluation of the Talent Transfer Initiative (Glazerman et al. 2013). 


8 Mean values for binary variables are proportions. For example, if Yi has a value of 1 for students who undermatched and 0 for students who did not, then 
the mean is the proportion of students-—or, if multiplied times 100, the percentage of students—in project j who undermatched. In other words, a mean of 
0.48 indicates that 48 percent of students in project j undermatched (.48 x 100 = 48). 

a As noted previously, some students are missing data for some characteristics measured before the lottery. In these cases, missing values were imputed 
using the dummy variable method. The dummy variable method involves substituting a constant value, such as 0, for all missing values of a given variable, 
and including a dummy variable with a value of “1” for cases with a missing value and a value of “O” for cases with a non-missing value. This method is 
consistent with the recommendation from the IES technical methods report What to Do When Data Are Missing in Group Randomized Controlled Trials (Puma 
et al. 2009) and is allowable under the standards of the What Works Clearinghouse”. 
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college they attended. To measure the mean difference in percentage points, the estimate is multiplied times 
100. For example, an estimate of .092 for y),can be multiplied times 100 (.092 x 100 = 9.2), which would indicate 
that the percentage of students in Find the Fit projects who undermatched is 9.2 percentage points higher than 
the percentage in regular UB advising projects. To test for impacts, the study conducted two-tailed t-tests at the 5 
percent level. Differences that did not meet this bar for statistical significance but were just short of it (p < .10) 
were consistently identified and noted as “promising.” Because the main measures of undermatch and selectivity 
fall into the same domain of college enrollment, the study did a Benjamini-Hochberg correction for multiple 
comparisons within this domain as a sensitivity test, the results are shown in Exhibit C.7b. 


The study also conducted exploratory analyses among students who enrolled in college to understand whether 
Find the Fit affected some characteristics of the colleges that students attended. Examining college characteristics 
only for students enrolled in college is the most straightforward and logical approach, but by excluding students 
who do not enroll in college, the analyses fall outside the lottery framework. Therefore, the study compared the 
baseline characteristics of students included in these analyses to confirm that the groups were similar before the 
lottery (see Exhibits B.19 through B.23). The study also examined results using the full group of students in the 
lottery (which required assigning an arbitrary value of O for the outcomes of students who were not enrolled in 
college) and found that the results were comparable. 


Covariates. All models controlled for random assignment block fixed effects to improve the precision of the 
estimated effects and to preserve the integrity of the random assignment because of the differing probabilities of 
random assignment to the Find the Fit or regular UB advising group within each block. In addition, the models 
controlled for baseline student-level covariates. Exhibit B.10 (above) summarized the covariates included in the 
estimation model. 


Treatment of Missing Data. As noted previously there were no missing data for selectivity of the college the 
student attended, persistence, transfer to a less selective college or dropout, or enrollment in a familiar college. 
So the full randomized sample was used to analyze the impact of Find the Fit on these college-going measures. 
This implies that no bias was introduced into the estimate of the effect of Find the Fit on these measures because 
of missing outcomes. Casewise deletion (that is, excluding students with missing outcomes) was used to account 
for missing data on all other college-going measures. 


The analyses included students with missing covariate values from the APR data or the baseline student survey. 
In these cases, missing values were imputed using the dummy variable method. The study replaced the missing 
covariate values with a placeholder (O) and created an indicator for the covariate having a missing value, which 
was included in the model. Simulations have shown that this approach to handling missing covariate data is 
likely to keep estimation bias at less than 0.05 standard deviations. 


Estimation Model for Subgroups 


In addition to examining the overall impacts of Find the Fit on student outcomes, the study also investigated the 
impact of Find the Fit within subgroups defined by student and project characteristics. Exhibit B.16 (above) 
described the rationale for examining each subgroup and the number of students in each subgroup. 


Models both (a) estimated the impact of Find the Fit for each of the subgroups and (b) tested for differences in 
impacts among categories of a subgroup indicator. Both types of results are reported in Section C; for example, 
impact estimates are reported for both male and female students, and the result is reported for a test of whether 
the magnitude of the impact for male students was different from the magnitude of the impact for female 
students. 


Because the study was designed to detect impacts for the full sample, not differences in impacts between 
subgroups, a difference in impacts between subgroups could only be detected when the true difference is 


Puma et al. (2009). 
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large.*° Because these tests are exploratory, multiple comparison adjustments were not made. It is important to 
note that with this approach, even if there were no significant differences, one might expect to detect as 
significant due to chance at least eight differences across the 144 tests conducted (9 outcomes x 16 subgroup 
categories). 


Student Subgroups. To address questions about impacts for subgroups of students, the subgroup variable was 
added to the level-1 (student-level) equation. For example: 


(3) ¥;; = Bo; + B1;(Female,;) + L4-2 Ba; (BaselineStudentCharacteristics,;) + &;; 


In addition, an interaction term between the subgroup variable and the treatment indicator was added to the 
level-2 (project-level) equation: 


(4) Boj = Yoo + YoiTreatment, + Yo2Historical Enrollment; + yoy Yon(RA_Blocks,,;) + Ho; 
(5) Bij = Vio t+ Y¥xiTreatment, 


In Equations (4) and (5) above, and using the indicator for female as an example subgroup indicator, y,, is the 
treatment impact for males, y,,; + y,, is the treatment impact for females, and y,, is the difference in the 
treatment impact between students who are female and students who are not. 


Project Subgroups. To address questions about impacts for subgroups of Upward Bound projects, the subgroup 
variable was added to the regression model as an interaction term with the treatment variable: 


(6) Bo; = Yoo + Yo1Treatment, + Yo2Historical Enrollment; + Y3-3 Yon(RA_Blocks,,;) + YoioRuralHostinst, * Trt; + 
Hoj 


In Equation (6), 7p, is the impact for non-rural host institutions, yp; + 7919 is the impact for rural host 
institutions, and yo, is the difference in impacts between projects hosted by rural and non-rural institutions. 
There is no main effect term for RuralHostiInst because that categorization is captured in the block dummies. 


Sensitivity Analyses 


Sensitivity analyses were conducted to examine whether the findings were affected by the particular statistical 
models used to estimate the effects of Find the Fit. The sensitivity analyses tested whether findings remained the 
same when the following three alternative statistical models were used: 


e Logistic regression models were estimated because logistic, rather than linear, regression is commonly 
used for binary outcomes. °° 


e A heteroscedasticity adjustment was included to further test the sensitivity of using linear models to 
estimate impacts for binary outcomes. 


e Models using the full randomized sample were re-estimated without covariate adjustment because 
randomization should yield treatment and control groups that are similar on both observed and 
unobserved characteristics, making covariate adjustment unnecessary.*! 


Findings from the sensitivity analyses were similar to findings from the main analyses (see Section C for details). 


42 Detailed information on the minimum detectable effect size for each outcome in the study is provided in section B.4.3 below. 

50 The logistic models, like the linear models, also account for the clustering of students within projects in the estimation of standard errors. This was 
accomplished using SAS Proc Glimmix. 

a The main analyses adjusted for baseline student characteristics in an effort to improve precision of the impact estimates, even though covariate adjustment 
was not necessary to account for baseline differences between treatment and control groups. For the analyses examining characteristics of colleges 
students attended, covariate adjustment is necessary because the sample is restricted to students enrolled in college. 
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Variation by Implementation Levels 


The study explored whether impacts on any of the main outcomes varied by projects’ levels of implementation 
of Find the Fit. As explained in the study’s first report, ** Find the Fit projects were categorized as high 
implementers if they implemented 75 percent or more of each Find the Fit component (personalized student 
folders, text messages, and training webinars for advisors); moderate implementers if they implemented more 
than a quarter but not necessarily 75 percent of each component; and low implementers if they implemented less 
than 25 percent of any one Find the Fit component. As implementation levels can only be observed for the Find 
the Fit projects it was necessary to predict implementation levels for the regular UB advising projects based on 
the relationship between project characteristics and the observed implementation level among Find the Fit 
projects. Known baseline characteristics of regular UB advising projects were used to predict whether the 
projects would be high, moderate, or low implementers, and the predicted implementation level was used for 
the subgroup analysis. The estimated implementation levels for the regular UB advising projects could differ 
from the actual levels if these projects had implemented Find the Fit. The study could have used predicted 
instead of the observed levels of implementation for Find the Fit projects to ensure that the estimation error was 
similar for both groups of projects. However, the study preferred not to use the predicted values when the actual 
values were known. 


Similar to the other subgroup analyses (described above), the models both estimated the impact of Find the Fit 
for each of the implementation levels and tested for differences in impacts among the levels. Find the Fit group 
percentage and impact are estimated using the study’s regression model. Results by implementation level for the 
main outcomes are shown in Section D. 

Exploratory Analyses of Relationship between Initial College Choices and Longer-Term College Outcomes 


The study also investigated descriptively whether the benefits of college quality and match identified in the 
literature hold true in this sample. Specifically, the study explored whether rates of persistence into the third 
year differed based on students’ initial college choices. These analyses explored descriptive patterns for the full 
sample regardless of whether students were in Find the Fit or regular UB advising projects. 


Ignoring treatment status, the study uses a two-level linear regression model to estimate the mean outcome for 
groups of students based on initial college choice, adjusting for the clustering of students within Upward Bound 
projects and for baseline covariates. 


The level-1 (student-level) equation is: 
s A 
Yi; = Boy + 2. By..; (InitialCollegeChoice Group,;) + y; Boa; (BaselineStudentCharacteristics,;) + &;; 
s=1 a=1 
Where all terms are defined the same as Equation (1), except as follows: 


Bo; is the covariate-adjusted mean value of the outcome in project j for the reference college-going 
group, such as students who did not attend college in the fall after high school. 


InitialCollegeChoiceGroup;; is the dummy variable or set of dummy variables representing the groups of 
students defined by initial college choices (for example dummy variables representing selectivity level of 
college attended, where not attending college is the reference group). 


fs; is the mean difference in the outcome for college-going group s compared to the reference group. 


The sum of £9; and £,,,; is the mean outcome for group s. 


In the level-2 (project-level) equation, the treatment indicator is omitted and all other terms are the same as in 
Equation (2): 
(7) Boj = Yoo + Yo2Historical Enrollment; + YA=3 Yon(RA_Blocksy;) + Mo; 


2 Martinez et al. 2018. 
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B.4.3 Power Analyses 
Power analyses determine 


e how large a sample is needed for a study to confidentially find a statistically significant estimated effect, 
given a specified true effect of the intervention. 


or, once the sample has been identified, 


e the minimum detectable effect (MDE) that a study will have an 80 percent chance to detect. 


This study was designed to detect a minimum true effect of 5 percentage points on the study’s primary outcome, 
undermatch, based on a targeted number of 200 participating Upward Bound projects and with an 80 percent 
probability of detecting a statistically significant effect at the 5 percent level. The study-achieved minimum 
detectable effect for undermatch was 6.3 percentage points. This implies that our study sample was powered to 
detect slightly larger effects than the design phase assumptions. 


The left column of Exhibit B.24 below shows the design phase assumptions and expected minimum detectable 
effect (MDE), whereas the right column shows the observed statistics and achieved MDE that the study, as 
conducted, had 80 percent power to detect on undermatch. The initial power analysis was based on specific 
assumptions about the number of students per project in the study sample, intraclass correlation, proportion of 
variance explained by covariates, and the success rate in the regular UB advising group. The actual power of the 
study differed slightly. That the achieved MDE was higher than the expected MDE for the undermatch outcome 
primarily resulted from fewer students in the regular UB advising group undermatching than the assumption 
made in the design phase. 


Exhibit B.24 Comparison of Design Assumptions and Sample Statistics for Undermatch 


! Observed Statistics in Study 
Design Phase Assumptions Sample 
Number of projects 200 194 
Proportion assigned to Find the Fit 50 51 
group 
Number of students per project 16 22 
Intraclass correlation 100 -082 
Student-level R-squared -080 .043 
Project-level R-squared 320 345 
Success rate? in regular UB advising 800 409 
group 
Standard Error of Impact Estimate 0.05 SD units, or 0.04 SD units, or 
2.0 percentage points 2.2 percentage points 

Minimum Detectable Effect § percentage polis 6a peroentage points 


SD is standard deviation. 
‘Proportion of students who undermatched. 
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The achieved MDEs for the study’s main outcomes—undermatch, selectivity level of the college attended, and 
persistence into the third fall after high school—ranged from 2 to 6 percentage points (Exhibit B.25). 


Exhibit B.25 Achieved Minimum Detectable Effects for Overall Sample 


Main Outcomes 


Undermatch (percentage points) 6.3 
Selectivity level of college that student attended (percentage points): 
Most competitive 1.5 
At least highly competitive 3.0 
At least very competitive 4.4 
At least competitive 6.3 
At least somewhat competitive 6.4 
Any four-year college 5.9 
Any college 5.1 
Persistence into or graduation by third fall after high school (percentage points) 5.0 
College entrance exam scores of incoming freshmen (SAT scores) 28.4 
Graduation rate (percentage points) 3.9 
Sticker price (dollars) 2,130.30 
Net price (dollars) 1,076.60 
Transfer to a less selective college or dropout by the second fall after high school 3.7 
(percentage points) 
Enrollment at host institution (percentage points) 6.2 
Distance from home (miles) 34.8 
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SECTION C. SUPPLEMENTAL TABLES AND INFORMATION ON STUDY 
FINDINGS 


This section includes statistical details on findings presented in the report as well as additional findings that are 
not in the report. The details are intended to supplement the findings presented in the report and provide the 
statistical information for readers interested in the technical details of the study’s findings. The underlying 
statistics presented in section were used to generate the exhibits on the effects of Find the Fit on college going 
found in the report and could be of interest to some readers. This information is reported for the outcome 
measures discussed in the report (“main” measures) as well as for the exploratory measures listed in Exhibit 
B.10. Additionally, the exhibits below provide the results for particular groups of students and Upward Bound 
projects (“subgroups”) as well as results from analyses examining the sensitivity (or robustness) of these effects 
to the statistical model chosen. 


In addition, this section contains the underlying statistics used to generate the exhibits on the relationship 
between the selectivity level of the college students initially attended and postsecondary progress for the full 
randomized sample, regardless of whether students received Find the Fit. The underlying statistics for exhibits 
about the relationship between attending a familiar college—that is, the Upward Bound host institution—and 
postsecondary progress are also included. 


C.1. Undermatch 


In the report, Exhibit 7 shows that Find the Fit had no effect on undermatch. Exhibit C.1 presents the estimated 
effects of Find the Fit on undermatch and corresponding p-values. It also shows that Find the Fit had no effect on 
undermatch for any of the exploratory analyses examining subgroups of students or projects. Exhibit C.2 below 
shows that Find the Fit had no effect on undermatch regardless of the statistical model used for analysis. 


Exhibit C.1 Effects on Undermatch, Overall and for Subgroups 


Overall Impact 


39.1 40.9 -1.7 (-6.1, 2.7) 2.2 441 
Gender 
Female 36.7 38.8 -2.1 (-7.0, 2.8) 2.5 .403 
Male 43.2 44.6 -1.4 (-7.2, 4.4) 3.0 .641 
F-test of difference” p=.816 
Race/Ethnicity 
Hispanic 36.3 40.4 -4.0 (11.5, 3.4) 3.8 .290 
White, non-Hispanic 47.6 51.4 -3.7 (-11.4, 3.9) 3.9 337 
Black, non-Hispanic 34.4 35.0 -0.6 (-7.0, 5.7) 3.2 .848 
Other, non-Hispanic 41.7 39.1 2.5 (-7.1, 12.1) 4.9 .606 
F-test of difference” p=.620 
College Entrance Exam Score Quartile 
Highest quartile 43.6 56.3 -12.6 (-25.9, 0.7) 6.8 -063 
Second quartile 42.9 46.4 -3.5 (-12.3, 5.4) 4.5 445 
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Third quartile 40.7 43.6 -2.9 (-10.3, 4.5) 3.8 .443 
Lowest quartile 37.8 40.6 -2.8 (-8.6, 3.1) 3.0 352 
Missing score 38.0 34.7 3.3 (-3.3, 9.9) 3.4 .329 
F-test of difference” p=.205 

Host Institution Type 

Four-year college 41.0 40.9 0.1 (-5.7, 5.9) 2.9 .969 
Two-year college 36.3 41.1 -4.8 (-13.1, 3.4) 4.2 .250 
Other 37.3 40.2 -3.0 (-15.9, 10.0) 6.6 .652 
F-test of difference’ p=.614 

Locale 

Rural 47.9 48.3 -0.4 (-13.3, 12.6) 6.6 .957 
City/suburb/town 38.0 39.9 -1.9 (-6.6., 2.8) 2.4 424 
F-test of difference” p=.824 


@ p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
» p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: Percentages represent the share of students who undermatched in their college choice. Find the Fit group percentages and impacts are estimated using the 
study’s regression model. 
Sample Sizes: 
Overall impact: Find the Fit group = 2,274 students, Regular UB advising group = 1,996 students. 
Gender: Find the Fit t group = 2,256 students, Regular UB advising group = 1,991 students. 
Race/ethnicity: Find the Fit group = 2,249 students, Regular UB advising group = 1,988 students. 
College entrance exam score quartiles: Find the Fit group = 2,274 students, Regular UB advising group = 1,996 students. 
Host institution type: Find the Fit group = 2,274 students, Regular UB advising group = 1,996 students. 
Locale: Find the Fit group = 2,274 students, Regular UB advising group = 1,996 students. 
Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline 
student survey 2015. 


Exhibit C.2. Sensitivity Analyses for Effects on Undermatch 


Linear regression (main model) 39.1 40.9 -1.7 6.1, 2.7) 2.2 441 
Logistic regression 38.4 40.9 -2.5 (-8.4, 3.5) 3.0 415 
Heteroscedasticity adjustment 39.1 40.9 -1.7 (-6.4, 2.9) 2.4 .463 
No covariates used in model 38.2 40.9 -2.7 (-7.7, 2.3) 2.5 284 


Notes: Percentage represents the share of students who undermatched in their college choice. Find the Fit group percentage and impact are estimated using the 
study’s regression model. 

Sample Size: Find the Fit group = 2,274 students, Regular UB advising group =1,996 students. 

Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline 
student survey 2015. 
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Undermatch can occur either because students enroll in a college that is less selective than the level to which 
they could be admitted or because they do not enroll in college at all. Not enrolling in college was the primary 
reason that students undermatched in this study. Among students who undermatched, almost three-quarters in 
both the Find the Fit projects group and regular UB advising projects group were not enrolled in college (Exhibit 
C.3). Only about 13 percent of undermatched students in Find the Fit projects and 15 percent of undermatched 
students in regular UB advising projects were enrolled in four-year colleges. 


Exhibit C.3 Among Undermatched Students, Whether and Where Students Enrolled 


ioraea paar Whether and where enrolled 
39% of Students Undermatched 41% of Students Undermatched 
0.9% 1.2% @ Not enrolled 


.. Two-year 
@ Non-Competitive/Somewhat 
Competitive four-year 


@ Competitive four-year 


@ Very competitive four-year 


Notes: Among students who undermatched, 653 Find the Fit group and 591 Regular UB advising group students were not enrolled, 112 Find the Fit group and 104 
Regular UB advising group students were enrolled in two-year colleges, 43 Find the Fit group and 30 Regular UB advising group students were enrolled in 
somewhat or non-competitive four-year colleges, 61 Find the Fit group and 81 Regular UB advising group students were enrolled in competitive four-year colleges, 
and 8 Find the Fit group and 10 Regular UB advising group students were enrolled in very competitive four-year colleges. 


Sample Size: Find the Fit group =877 students who were undermatched, Regular UB advising group =816 students who were undermatched. 
Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline 
student survey 2015. 


This study’s approach to measuring undermatch was modeled after prior research, * but the study also 
conducted exploratory analyses that examined whether the findings would change if an alternative approach to 
measuring undermatch was used to examine the robustness of the finding. This alternative approach to 
measuring undermatch compared a student’s college entrance exam scores to those of other incoming freshmen 
at the college she or he attended to assess how well the student fit academically among her or his peers.** The 
sample used to measure undermatch with this approach is more restricted because some colleges, including 
most two-year colleges, do not require college entrance exam scores, which means that this measure of 
undermatch could not be constructed for students at these colleges. Regardless, the overall finding was the same 
as for the “main” undermatch measure: Find the Fit had no effect on this alternative measure of undermatch. 
Exhibit C.4 below also shows that Find the Fit had no effect on this alternative measure of undermatch for any of 
the exploratory analyses that examined subgroups of students or projects. Exhibit C.5 below shows that Find the 
Fit had no effect on this exploratory measure of undermatch regardless of the statistical model used for analysis. 


= Smith, Pender, and Howell 2013. 
Howell, Pender, and Kumar 2016; Page and Iriti 2016. 
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Exhibit C.4 Differences in Undermatch Based on College Entrance Exam Scores of Incoming 
Freshmen, Overall and for Subgroups 


Overall Difference 


51.4 55.2 -3.8 (-8.4, 0.9) 2.4 110 
Gender 
Female 48.8 53.0 -4,2 (-9.5, 1.0) 2.7 3 
Male 56.0 58.7 -2.6 (-8.9, 3.6) 3.2 .408 
F-test of difference’ p=.640 
Race/Ethnicity 
Hispanic 47.8 55.3 -7.5 (-15.4, 0.4) 4.0 .062 
White, non-Hispanic 59.1 63.4 -4.3 (-12.6, 4.0) 4.2 310 
Black, non-Hispanic 51.1 54.3 -3.2 (-10.0, 3.6) 3.5 .352 
Other, non-Hispanic 44.8 40.0 4.8 (-5.8, 15.4) 5.4 375 
F-test of difference? p= .267 
College Entrance Exam Score Quartile 
Highest quartile 54.6 57.4 -2.9 (-15.6, 9.9) 6.5 .660 
Second quartile 35.7 42.5 -6.8 (-15.6, 1.7) 4.4 118 
Third quartile 47.3 47.9 -0.6 (-7.7, 6.5) 3.6 .874 
Lowest quartile 57.5 62.4 -4.9 (-10.5, 0.6) 2.8 .082 
F-test of difference’ p=.603 
Host Institution Type 
Four-year college 49.0 53.4 -4.4 (-10.4, 1.7) 3.1 156 
Two-year college 59.8 62.4 -2.6 (-11.8, 6.5) 4.7 574 
Other 48.0 51.6 -3.6 (-16.7, 9.5) 6.7 591 
F-test of difference’ p=-952 
Locale 
Rural 69.0 66.0 3.0 (-11.2, 17.2) 7.2 .681 
City/suburb/town 49.4 54.0 -4.6 (-9.5, 0.3) 2.5 .067 
F-test of difference’ p=.322 


3 p-Values shown in this column are for tests of whether there was a statistically significant difference for the subgroup category in the row. 
> p-Values shown in this row are for a test of statistical differences between the categories of the subgroup in the rows above. 
Notes: Differences were calculated on the sample of students who enrolled in the first year. Percentage represents the share of students who undermatched in 
their college choice. Find the Fit group percentages and differences are estimated using the study’s regression model. 
Sample Sizes: 

Overall difference: Find the Fit group = 1,738 students, Regular UB advising group = 1,489 students. 

Gender: Find the Fit group = 1,725 students, Regular UB advising group = 1,486 students. 

Race/ethnicity: Find the Fit group = 1,719 students, Regular UB advising group = 1,484 students. 

College entrance exam score quartiles: Find the Fit group = 1,738 students, Regular UB advising group = 1,489 students. 

Host institution type: Find the Fit group = 1,738 students, Regular UB advising group = 1,489 students. 

Locale: Find the Fit group = 1,738 students, Regular UB advising group = 1,489 students. 
Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2013-14, 2014-15; IPEDS 2015-16; baseline student survey 2015. 
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Exhibit C.5 Sensitivity Analyses for Difference in Undermatch Based on College Entrance Exam 
Scores of Incoming Freshmen 


Find the Fit Regular UB Estimated 
Group Advising Difference (95% 
Students Group Confidence Standard 
| Model (%) Students (%) Interval) Error p-Value | 
Linear regression (main model) 51.4 55.2 -3.8 (-8.4, 0.9) 2.4 110 
Logistic regression 49.6 55.2 -5.6 (-12.3, 1.1) 3.4 101 
Heteroscedasticity adjustment 51.4 55.2 -3.8 (-8.7, 1.2) 2.5 133 


Notes: Differences were calculated on the sample of students who enrolled in the first year. Percentage represents the share of students who undermatched in 
their college choice. Find the Fit group percentage and difference are estimated using the study’s regression model 

Sample Size: Find the Fit group = 1,738 students, Regular UB advising group = 1,489 students. 

Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2013-14, 2014-15; IPEDS 2015-16; baseline student survey 2015. 


C.2 Selectivity Level of the College Attended 


In the report, Exhibit 8 shows that Find the Fit increased student attendance at colleges at the highest selectivity 
levels. Exhibit C.6 presents the estimated effects of Find the Fit on attendance at colleges at each selectivity level 
and corresponding p-values. Exhibit C.7a shows that the results of Find the Fit on attendance at colleges at 
various selectivity levels were consistent regardless of the statistical model used for analysis. Exhibit C.7b shows 
that the results of Find the Fit on undermatch and attendance at colleges at various selectivity levels were 
consistent after conducting a Benjamini-Hochberg correction for multiple comparisons in the enrollment 
domain. 


For the exploratory analyses examining subgroups, positive effects of Find the Fit on the selectivity level of 
colleges that students attended were found for eight of the 16 subgroups examined, and there were no significant 
negative effects for any groups (Exhibits C.8 and C.9 below). For example, students in Find the Fit projects who 
were female or Black, non-Hispanic were 4 percentage points more likely than their counterparts in regular UB 
advising projects to enroll in a college that was at least highly competitive (Exhibit C.8). Likewise, anong Upward 
Bound projects hosted by two-year colleges and those located in non-rural areas, students whose projects had 
access to Find the Fit were 4 percentage points more likely than their counterparts whose projects offered 
regular UB advising to attend a college that was at least highly competitive (Exhibit C.9). Also, positive effects on 
college selectivity were found not just for the most academically prepared students but for those with middle-of- 
the-road test scores. 


Exhibit C.6 Effects on the Selectivity of the College Attended in the First Fall after High School 


Regular UB Estimated 
Find the Fit Advising Impact (95% 
Group Group Students Confidence Standard 
Selectivity Level Students (%) (%) Interval) Error p-Value | 
Most competitive 2.2 1.3 0.9 (-0.2, 1.9) 0.5 .096 
At least highly competitive 6.9 3.7 3.2 (1.1, 5.3) 1.1 .003 
At least very competitive 16.4 13.1 3.3 (0.2, 6.4) 1.6 .037 
(At least competitive 38.8 37.8 1.0 (-3.4, 5.4) ye) .650 
At least somewhat competitive 46.5 44.0 2.5 (-1.9, 7.0) 2.3 .266 
Any four-year college 50.0 51.1 -1.1 (-5.2, 3.0) 2.1 597 
Any college 70.6 72.0 -1.4 (-5.0, 2.2) 1.8 454 


Notes: Percentage represents the share of students who attended a college of at least a given selectivity level. For example, “at least very competitive” includes 
attending colleges at the two selectivity levels above very competitive: highly competitive and most competitive. Differences were compared at each level by 
combining students who had attended colleges at that selectivity level and the levels above. Find the Fit group percentage and impact are estimated using the 
study’s regression model. 

Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline 
student survey 2015. 
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Exhibit C.7a_ Sensitivity Analyses for Effects on the Selectivity of the College Attended in the First Fall 
after High School 


Most Competitive 


Linear regression (main model) Dd) 1.3 0.9 (-0.2, 1.9) 0.5 .096 
Logistic regression 2.3 1.3 1.0 (0.1, 1.9) 0.4 .029 
Heteroscedasticity adjustment 2.2 1.3 0.9 (-0.2, 2.0) 0.6 110 
No covariates used in model 2.5 1.3 1.1 (-0.0, 2.3) 0.6 .053 
Linear regression (main model) 6.9 3.7 3.2 (1.1, 5.3) 1.1 .003 
Logistic regression 6.7 3.7 3.0 (1.3, 4.7) 0.9 .OO1 
Heteroscedasticity adjustment 6.9 3.7 3.2 (1.1, 5.3) 1.1 .003 
No covariates used in model 7.3 3.7 3.6 (1.2, 6.0) 1.2 .003 
Linear regression (main model) 16.4 13.1 3.3 (0.2, 6.4) 1.6 .037 
Logistic regression 17.5 13.1 4.4 (1.3, 7.5) 1.6 .006 
Heteroscedasticity adjustment 16.4 13.1 3.3 (0.1, 6.4) 1.6 .043 
No covariates used in model 17.4 13.1 4.3 (0.5, 8.2) 2.0 .028 
Linear regression (main model) 38.8 37.8 1.0 (3.4, 5.4) 2.2 .650 
Logistic regression 40.5 37.8 2.6 (-4.6, 9.8) 3.7 473 
Heteroscedasticity adjustment 38.8 37.8 1.0 (-3.6, 5.6) 2.3 .665 
No covariates used in model 40.5 37.8 2.7 (-2.9, 8.3) 2.9 350 
Linear regression (main model) 46.5 44.0 2.5 (-1.9, 7.0) 2.3 .266 
Logistic regression 48.6 44.0 4.6 (-2.9, 12.0) 3.8 .227 
Heteroscedasticity adjustment 46.5 44.0 2.5 (-2.2, 7.2) 2.4 .288 
No covariates used in model 48.4 44.0 4.4 (-1.3, 10.1) 2.9 132 
Linear regression (main model) 50.0 51.1 -1.1(-5.2, 3.0) 2.1 597 
Logistic regression 51.8 51.1 0.8 (-2.2, 3.8) 1.5 .614 
Heteroscedasticity adjustment 50.0 51.1 -1.1 (-5.4, 3.2) 2.2 617 
No covariates used in model 52.0 51.1 0.9 (-4.3, 6.1) 2.6 .735 
Linear regression (main model) 70.6 72.0 -1.4 (-5.0, 2.2) 1.8 .454 
Logistic regression 72.0 72.0 0.1 (--0.1, 0.3) 0.1 391 
Heteroscedasticity adjustment 70.6 72.0 -1.4 (-5.2, 2.5) 2.0 485 
No covariates used in model 71.8 72.0 -0.1 (-4.7, 4.4) 2.3 .953 


Notes: Percentage represents the share of students who attended a college of at least a given selectivity level. For example, “at least very competitive” includes 
attending colleges at the two selectivity levels above very competitive: highly competitive and most competitive. Differences were compared at each level by 
combining students who had attended colleges at that selectivity level and the levels above. Find the Fit group percentages and impacts are estimated using the 
study’s regression model. Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline 
student survey 2015. 
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Exhibit C.7b Benjamini-Hochberg correction for multiple comparisons in the enrollment domain 
(Undermatch and Selectivity Level) 


0.003 1 0.025 Yes Yes 
0.037 2 0.05 Yes Yes 


Sample Size (Undermatch): Find the Fit group = 2,274 students, Regular UB advising group = 1,996 students. 

Sample Size (Selectivity): Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline 
student survey 2015. 


Exhibit C.8 —_ Effects on the Selectivity of the College Attended in the First Fall after High School, by 
Student Subgroup 


Most Competitive 


Gender 

Female 2.8 1.3 1.5 (0.3, 2.7) 0.6 .O16 
Male 1) 1.3 -0.1 (-1.7, 1.3) 0.8 .856 
F-test of difference’ p=.047 

Race/Ethnicity 

Hispanic 3.1 1.4 1.7 (0.2, 3.5) 1.0 .083 
White, non-Hispanic 1.1 1.0 0.2 (-1.9, 2.0) 1.0 .878 
Black, non-Hispanic 2.0 0.5 1.5 (-0.1, 3.1) 0.8 .065 
Other, non-Hispanic 3.3 4.4 -1.0 (-3.6, 1.5) 1.3 .423 
F-test of difference” p=.229 

College Entrance Exam Score Quartile 

Highest quartile 17.0 11.9 5.1 (1.5, 8.6) 1.8 .005 
Second quartile 4.7 3.7 1.0 (-1.4, 3.3) 1.2 .426 
Third quartile 2.0 0.8 1.2 (-0.7, 3.1) 1.0 .210 
Lowest quartile 0.6 0.3 0.4 (-1.1, 1.8) 0.7 .604 
Missing score 1.0 0.3 0.7 (-1.0, 2.3) 0.9 425 
F-test of difference? p=.169 

Gender 

Female 8.0 3.7 4.3 (2.0, 6.6) 1.2 .OOO 
Male 5.0 3.8 1.2 (-1.5, 3.8) 1.4 384 
F-test of difference? p=.016 

Race/Ethnicity 

Hispanic 7.3 4.3 3.0 (-0.3, 6.3) 1.7 .078 
White, non-Hispanic 5.6 3.6 2.0 (-1.5, 5.5) 1.8 271 
Black, non-Hispanic 5.6 1.6 4.0 (1.1, 7.0) 1.5 .007 
Other, non-Hispanic 12.9 9.5 3.4 (-0.9, 7.7) Deed 9 
F-test of difference? p=.810 
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College Entrance Exam Score Quartile 

Highest quartile 30.8 19.8 11.0 (5.2, 16.8) 3.0 .000 
Second quartile 18.2 12.3 5.8 (0.1, 6.8) 2.0 .004 
Third quartile 6.6 3.2 3.4 (0.1, 6.8) 1.7 .042 
Lowest quartile 2.2 1.2 1.1 (1.6, 3,7) 1.3 422 
Missing score 4.8 1.3 3.5 (0.5, 6.4) 1.5 .022 
F-test of difference” p=.009 

Gender 

Female 17.2 13.3 4.0 (0.6, 7.4) 1.7 .022 
Male 14.9 12.9 2.1 (-1.9, 6.1) 2.0 312 

F-test of difference” p=.34l 

Race/Ethnicity 

Hispanic 18.3 13.9 4.4 (-0.6, 9.5) 2.6 .083 
White, non-Hispanic 14.1 12.4 1.7 (-3.6, 7.0) 2.7 528 
Black, non-Hispanic 12.9 9.4 3.5 (-0.9, 7.9) 2.2 114 

Other, non-Hispanic 27.3 24.0 3.3 (-3.2, 9.9) 3.3 .316 

F-test of difference’ p=.880 

College Entrance Exam Score Quartile 

Highest quartile 51.7 38.6 13.1 (4.1, 22.0) 4.6 .004 
Second quartile 40.3 34.2 6.2 (0.1, 12.2) 3.1 .045 
Third quartile 19.2 14.6 4.6 (-0.5, 9.6) 2.6 .O77 
Lowest quartile 7.6 6.3 1.3 (-2.7, 5.3) 2.0 515, 

Missing score 10.5 8.2 2.3 (-2.2, 6.8) 2.3 313 

F-test of difference’ p=.099 

Gender 

Female 41.4 40.1 1.3 (3.5, 6.1) 2.4 593 
Male 34.5 34.0 0.4 (-5.1, 6.0) 2.8 .878 
F-test of difference? p=.744 

Race/Ethnicity 

Hispanic 39.2 33.7 5.4 (-1.5, 12.3) 3.5 124 

White, non-Hispanic 40.4 39.6 0.8 (-6.4, 8.1) 3.7 .820 
Black, non-Hispanic 35.4 37.4 -2.0 (-8.1, 4.1) 3.1 528 
Other, non-Hispanic 44.7 42.9 1.8 (-7.1, 10.6) 4.5 .699 
F-test of difference’ p=.364 

College Entrance Exam Score Quartile 

Highest quartile 79.4 70.3 9.1 3.0, 21.1) 6.1 139 

Second quartile 67.0 63.0 4.0 (-4.2, 12.2) 4.2 339 
Third quartile 44.1 47.7 -3.6 (-10.6, 3.3) 3.5 .302 
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Lowest quartile 268  ©=©— 23.9 29(26,85) 28) 301 


Missing score 33.0 34.0 -1.1 (-7.3, 5.2) 3.2 .738 
F-test of difference’ p=.165 

Gender 

Female 49.8 47.1 2.6 (-2.2, 7.5) 2.5 .288 
Male 41.0 38.8 2.3 (-3.4, 7.9) 2.9 429 
F-test of difference’ p=.891 

Race/Ethnicity 

Hispanic 45.7 38.7 7.0 (-0.0, 14.1) 3.6 .051 
White, non-Hispanic 47.3 44.6 2.7 (-4.7, 10.1) 3.8 469 
Black, non-Hispanic 46.1 46.2 -O.2 (-6.4, 6.1) 3.2 .960 
Other, non-Hispanic 47.1 45.8 1.3 (-7.8, 10.3) 4.6 .783 
F-test of difference? p=.385 

College Entrance Exam Score Quartile 

Highest quartile 82.0 73.3 8.8 (-3.5, 21.0) 6.2 160 
Second quartile 73.1 67.9 5.2 (-3.1, 13.6) 4.2 .217 
Third quartile 54.2 56.0 -1.8 (-8.9, 5.2) 3.6 .614 
Lowest quartile 36.2 31.5 4.8 (-0.9, 10.4) 2.9 .099 
Missing score 38.4 38.2 0.3 (-6.1, 6.6) 3.2 .936 
F-test of difference” p=.238 

Gender 

Female 53.1 53.5 -0.4 (-5.0, 4.1) 2.3 .884 
Male 44.6 47.0 -2.4 (-7.8, 2.9) oll .374 
F-test of difference’ p=.444 

Race/Ethnicity 

Hispanic 50.7 50.3 0.4 (-6.6, 7.0) 3.5 .903 
White, non-Hispanic 49.4 48.8 0.6 (-6.4, 7.7) 3.6 .859 
Black, non-Hispanic 49.2 51.7 -2.5 (-8.3, 3.4) 3.0 .410 
Other, non-Hispanic 51.7 54.9 -3.2 (-12.0, 5.6) 4.5 .483 
F-test of difference” p=.801 

College Entrance Exam Score Quartile 

Highest quartile 82.6 78.2 4.4 (-7.9, 16.6) 6.2 .480 
Second quartile 73.6 71.2 2.4 (-5.8, 10.6) 4.2 565 
Third quartile 56.5 60.5 -4.0 (-10.9, 2.9) 3.5 251 
Lowest quartile 40.2 39.7 0.4 (5.1, 5.7) Doll .873 
Missing score 43.5 47.0 -3.5 (-9.6, 2.5) 3.1 .257 
F-test of difference” p=.423 
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Any College 


Gender 

Female 73.0 74.4 -1.4 (-5.5, 2.6) 2.1 .496 
Male 66.6 67.7 -1.0 (-5.9, 3.9) 2.5 .681 
F-test of difference” p=.885 

Race/Ethnicity 

Hispanic 71.7 72.5 -0.8 (-7.0, 5.4) 3.2 .803 
White, non-Hispanic 68.5 73.2 -4.8 (-11.2, 1.7) Bos} 149 
Black, non-Hispanic 71.6 70.9 0.7 (-4.6, 6.0) 2.7 .797 
Other, non-Hispanic 70.2 71.6 -1.4 (-9.7, 6.8) 4.2 .733 
F-test of difference’ p=.610 

College Entrance Exam Score Quartile 

Highest quartile 86.5 94.1 -7.6 (-19.2, 4.1) 5.9 .202 
Second quartile 83.7 83.1 0.6 (-7.1, 8.3) 3.9 .887 
Third quartile 73.7 77.2 -3.5 (-9.9, 2.9) 3.3 .279 
Lowest quartile 65.2 64.0 1.2 (-3.7, 6.1) Ds) .631 
Missing score 67.7 70.7 -3.0 (-8.6, 2.6) 2.8 .289 
F-test of difference’ p=.429 


3 p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
> p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: Percentage represents the share of students who attended a college of at least a given selectivity level. For example, “at least very competitive” includes 
attending colleges at the two selectivity levels above very competitive: highly competitive and most competitive. Differences were compared at each level by 
combining students who had attended colleges at that selectivity level and the levels above. Find the Fit group percentages and impacts are estimated using the 
study’s regression model. 
Sample Sizes: 

Gender: Find the Fit group = 2,318 students, Regular UB advising group = 2,102 students. 

Race/ethnicity: Find the Fit group = 2,311 students, Regular UB advising group = 2,099 students. 

College entrance exam score quartiles: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 
Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline 
student survey 2015. 


Exhibit C.9 Effects on the Selectivity of the College Attended in the First Fall after High School, by 
Project Subgroup 


Most Competitive 


Host Institution Type 

Four-year college 2.2 1.3 0.9 (-0.5, 2,3) 0.7 .206 
Two-year college 1.1 0.2 0.9 (-1.2, 2.8) 1.0 376 
Other 4.9 3.8 1.1 (-1.9, 4.3) 1.6 488 
F-test of difference” p=.992 

Locale 

Rural 0.5 0.0 0.5 (-2.7, 3.7) 1.6 .749 
City/suburb/town 2.4 1.5 0.9 (-0.2, 2.0) 0.6 .099 
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F-test of difference” p=.801 

Host Institution Type 

Four-year college 6.4 3.6 2.7 (-0.0, 5.4) 1.4 -052 
Two-year college 6.2 1.9 4.3 (0.4, 8.2) 2.0 .031 
Other 10.7 7.9 2.8 (-3.4, 9.0) 3.2 .372 
F-test of difference? p=.804 

Locale 

Rural 1.8 1.3 0.6 (-5.6, 6.7) 3.1 .858 
City/suburb/town 7.6 4l 3.5 (1.3, 5.7) 1.1 .002 
F-test of difference? p=.376 

Host Institution Type 

Four-year college 18.0 14.7 3.3 (-0.8, 7.3) 2.1 All 

Two-year college 11.5 8.6 2.9 (-2.9, 8.7) 3.0 328 
Other 18.1 13.9 4.2 (-5.0, 13.3) 4.7 .370 
F-test of difference” p=.974 

Locale 

Rural 6.3 2S) 3.4 (-5.8, 12.5) 4.7 .470 
City/suburb/town 17.7 14.4 3.3 (0.0, 6.6) 1.7 .052 
F-test of difference” p=.984 

Host Institution Type 

Four-year college 41.0 A1.6 -0.5 (-6.3, 5.2) 2.9 -855 
Two-year college 32.3 29.5 2.8 (-5.5, 11.0) 4.2 .509 
Other 40.1 35.7 4.4 (-8.6, 17.4) 6.6 .509 
F-test of difference” p=.698 

Locale 

Rural 30.6 23.4 7.2 (-5.8, 20.1) 6.6 .277 
City/suburb/town 39.9 39.7 0.2 (-4.5, 4.9) 2.4 .931 
F-test of difference? p=.321 

Host Institution Type 

Four-year college 50.8 47.6 3.1 (-2.8, 9.0) 3.0 .297 
Two-year college 35.9 32.8 3.2 (-5.2, 11.6) 4.3 .456 
Other 46.5 48.1 -1.6 (-14.9, 11.7) 6.8 .812 
F-test of difference? p=.806 

Locale 

Rural 31.1 29.7 1.4 (-11.8, 14.7) 6.8 .832 
City/suburb/town 48.5 45.8 2.7 (-2.1, 7.5) 2.4 .270 
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F-test of difference 


 -p=.861 


Host Institution Type 

Four-year college 53.5 57.5 -4.0 (-9.4, 1.3) 2.7 144 
Two-year college 39.9 35.0 4.8 (-2.9, 12.5) 3.9 221 
Other 49.9 50.4 -0.5 (-12.7, 11.5) 6.2 .933 
F-test of difference? p=.185 

Locale 

Rural 34.0 35.6 -1.6 (-13.7, 10.7) 6.2 .801 
City/suburb/town 52.0 53.0 -1.0 (-5.4, 3.3) 2.2 .659 
F-test of difference” p=.930 

Host Institution Type 

Four-year college 69.4 72.8 -3.4 (-8.0, 1.3) 2.4 157 
Two-year college 71.2 72.0 -0.8 (-7.6, 6.1) 3.5 .826 
Other 74.5 67.7 6.8 (-3.7, 17.4) 5.4 .204 
F-test of difference” p=.216 

Locale 

Rural 61.6 66.9 -5.3 (-16.0, 5.4) 5.5 .329 
City/suburb/town 71.7 72.6 -0.9 (-4.7, 3.0) 2.0 .658 
F-test of difference’ p=.442 


* p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 

> p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 

Notes: Percentage represents the share of students who attended a college of at least a given selectivity level. For example, “at least very competitive” includes 
attending colleges at the two selectivity levels above very competitive: highly competitive and most competitive. Differences were compared at each level by 
combining students who had attended colleges at that selectivity level and the levels above. Find the Fit group percentages and impacts are estimated using the 
study’s regression model. 

Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline 
student survey 2015. 


C.2.1 Selectivity Level of the College Attended in the Second Fall and Third Fall after 
High School 


The report discusses the exploratory analyses of the effects of Find the Fit on the selectivity of colleges that 
students attended two and three years after high school. Exhibit C.10 shows differences in the selectivity level of 
the colleges that students attended two years after high school, and Exhibit C.11 below shows differences three 
years after high school. These exhibits also show that the effects of Find the Fit on the selectivity of colleges that 
students attended in the second and third fall after high school remain the same regardless of the statistical 
model used for analysis. 


Exhibit C.10 Analyses for Effects on the Selectivity of the College Attended in the Second Fall after 


High School 


Most Competitive 


Linear regression (main model) 
Logistic regression 
Heteroscedasticity adjustment 
No covariates used in model 


Linear regression (main model) 
Logistic regression 
Heteroscedasticity adjustment 
No covariates used in model 


Linear regression (main model) 
Logistic regression 
Heteroscedasticity adjustment 
No covariates used in model 


Linear regression (main model) 
Logistic regression 
Heteroscedasticity adjustment 
No covariates used in model 


Linear regression (main model) 
Logistic regression 
Heteroscedasticity adjustment 
No covariates used in model 


Linear regression (main model) 
Logistic regression 
Heteroscedasticity adjustment 
No covariates used in model 


Linear regression (main model) 
Logistic regression 
Heteroscedasticity adjustment 
No covariates used in model 


2.3 1.4 0.9 (-0.2, 2.0) 0.5 .099 
2.4 1.4 1.0 (0.1, 1.9) 0.5 .034 
2.3 1.4 0.9 (-0.2, 2.0) 0.6 117 
2.5 1.4 1.1 (0.0, 2.3) 0.6 .055 
7.0 3.7 3.3 (1.2, 5.4) 1.1 .002 
6.8 Sol 3.1 (1.4, 4.8) 0.9 .000O 
7.0 3.7 3.3 (1.2, 5.4) 1.1 .002 
7.4 Bol 3.7 (1.3, 6.1) 1.2 .003 
15.1 11.8 3.3 (0.4, 6.2) 1.5 .025 
16.1 11.8 4.3 (1.5, 7.2) 1.5 .003 
15.1 11.8 3.3 (0.3, 6.3) 1.5 .031 
16.0 11.8 4.2 (0.5, 7.9) 1.9 .024 
34.1 32.7 1.4 (-2.5, 5.2) 2.0 488 
35.3 32.7 2.6 (-2.3, 7.5) 2.5 304 
34.1 32.7 1.4 (-2.6, 5.3) 2.0 .499 
35.5 : 2.8 (-2.3, 7.9) : : 
40.8 37.7 3.1 0.8, 6.9) 2.0 116 
42.7 37.7 5.0 (-0.7, 10.7) 2.9 .084 
40.8 37.7 3.1 0.9, 7.0) 2.0 128 
42.4 37.7 4.7 (-0.4, 9.7) 2.6 .073 
45.0 44.1 0.9 (-2.9, 4.6) 1.9 .653 
47.3 44.1 3.2 (-7.8, 14.2) 5.6 573 
45.0 44.1 0.9 (-3.1, 4.8) 2.0 .670 
46.6 44.1 2.5 (-2.4, 7.4) 2.5 317 
62.4 62.3 0.2 (-3.4, 3.7) 1.8 .930 
64.4 62.3 2.2 (-53.4, 57.8) 28.4 .939 
62.4 62.3 0.2 (-3.6, 3.9) 1.9 .934 
63.6 62.3 1.3 (-3.4, 6.0) 2.4 583 


Notes: Percentage represents the share of students who attended a college of at least a given selectivity level. For example, “at least very competitive” includes 
attending colleges at the two selectivity levels above very competitive: highly competitive and most competitive. Differences were compared at each level by 
combining students who had attended colleges at that selectivity level and the levels above. Find the Fit group percentages and impacts are estimated using the 


study’s regression model. 


Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 


Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; 


IPEDS 2015-16; baseline student survey 2015. 
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Exhibit C.11 Analyses for Effects on the Selectivity of the College Attended in the Third Fall after 
High School 


Most Competitive 


Linear regression (main model) ie} 1.3 1.0 (0.0, 2.1) 0.5 .056 
Logistic regression 2.4 1.3 1.1 (0.1, 2.1) 0.5 .025 
Heteroscedasticity adjustment 2.3 1.3 1.0 (-0.1, 2.1) 0.6 .067 
No covariates used in model 2.5 1.3 1.2 (0.1, 2.4) 0.6 .033 
Linear regression (main model) 6.7 3.6 3.2 (1.1, 5.2) 1.0 .002 
Logistic regression 6.5 3.6 2.9 (1.3,4.5) 0.8 .000 
Heteroscedasticity adjustment 6.7 3.6 3.2 (1.1, 5.2) 1.1 .003 
No covariates used in model 71 3.6 3.5 (1.2, 5.9) 1.2 .003 
Linear regression (main model) 14.6 11.6 3.0 (0.1, 5.9) 1.5 .044 
Logistic regression 15.5 11.6 3.9 (1.1, 6.7) 1.4 .007 
Heteroscedasticity adjustment 14.6 11.6 3.0 (0.0, 6.0) 1.5 .052 
No covariates used in model 15.5 11.6 3.9 (0.2, 7.5) 1.8 .037 
Linear regression (main model) 31.1 30.6 0.4 (-3.3, 4.2) 1.9 815 

Logistic regression 32.1 30.6 1.5 (-3.5, 6.4) 2.5 568 
Heteroscedasticity adjustment 31.1 30.6 0.4 (-3.4, 4.3) 2.0 .823 
No covariates used in model 32.6 30.6 2.0 (-2.8, 6.8) 2.5 .420 
Linear regression (main model) 37.0 34.5 2.5 (-1.2, 6.2) 1.9 181 

Logistic regression 38.7 34.5 4.2 (-1.2, 9.6) 2.8 .128 

Heteroscedasticity adjustment 37.0 34.5 2.5 (-1.3, 6.3) 2.0 198 

No covariates used in model 38.6 34.5 4.1 (-0.7, 8.9) 2.4 .O91 
Linear regression (main model) 40.8 39.8 1.0 (-2.7, 4.7) 1.9 .596 
Logistic regression 42.7 39.8 3.0 (-5.2, 11.1) Al .476 
Heteroscedasticity adjustment 40.8 39.8 1.0 (-2.9, 4.9) 2.0 612 

No covariates used in model 42.5 39.8 2.7 (-2.1, 7.5) 2.4 .266 
Linear regression (main model) 54.2 By) 2.0 (-1.7, 5.6) 1.9 .294 
Logistic regression 55.7 52.2 3.4 (-2.8, 9.7) 3.2 .282 
Heteroscedasticity adjustment 54.2 52.2 2.0 (-1.9, 5.8) 2.0 317 

No covariates used in model 55.7 52.2 3.5 (-1.3, 8.3) 2.5 155 


Notes: Percentage represents the share of students who attended a college of at least a given selectivity level. For example, “at least very competitive” includes 
attending colleges at the two selectivity levels above very competitive: highly competitive and most competitive. Differences were compared at each level by 
combining students who had attended colleges at that selectivity level and the levels above. Find the Fit group percentages and impacts are estimated using the 
study’s regression model. 

Sample Size: Find the Fit group =2,336 students, Regular UB advising group =2,107 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; NCES-Barron’s Admissions Competitiveness Index 2014; college entrance exam score data 2015; IPEDS 2015- 
16; baseline student survey 2015. 
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C.2.2 Alternative Definitions of Quality 


In addition to examining whether Find the Fit affected the quality of the colleges students attended as measured 
by college selectivity level, the study also conducted exploratory analyses to examine whether there were 
differences in college quality using two alternative measures. The first alternative was the college entrance exam 
scores of incoming freshmen. This alternative examines the academic qualifications of students who choose to 
attend a given college because students with better academic qualifications can be taught at a rigorous level, and 
as such can be viewed as a measure of college quality. The second alternative was the college’s graduation rate. A 
college’s graduation rate provides another partial measure of quality because it reflects the instruction and 
supports provided by the college to help students graduate while also reflecting the characteristics of students 
who attend the college. 


For the first alternative measure, college entrance exam scores of incoming freshmen, there was no significant 
difference in the quality of colleges attended by students in Find the Fit projects versus regular UB advising 
projects (Exhibit C.12 below). Similarly, there was no significant difference on this measure of college quality for 
any student or project subgroups examined. 


Exhibit C.12 Differences between College Entrance Exam Scores of Incoming Freshmen at the 
Colleges Attended by Find the Fit and Regular UB Advising Group Students, Overall and 
for Subgroups 


Overall Difference 


1,172.7 1,166.1 6.6 (-13.2, 26.5) 10.1 513 
Gender 
Female 1,174.1 1,163.8 10.2 (-10.8, 31.4) 10.8 342 
Male 1,169.2 1,171.0 -1.8 (-26.7, 23.2) 12.7 .890 
F-test of difference” p=.287 
Race/Ethnicity 
Hispanic 1,199.5 1,189.7 9.7 (-20.9, 40.4) 15.6 534 
White, non-Hispanic 1,166.8 1,165.9 0.8 (-31.8, 33.5) 16.6 .959 
Black, non-Hispanic 1,133.9 1,117.5 16.4 (-10.2, 43.0) 13.5 .226 
Other, non-Hispanic 1,239.5 1,262.0 -22.5 (-59.8, 14.8) 19.0 .238 
F-test of difference” p=.293 
College Entrance Exam Score Quartile 
Highest quartile 1,314.1 1,276.3 37.8 (-2.6, 78.2) 20.6 .067 
Second quartile 1,237.5 1,232.8 4.7 (-25.9, 35.3) 15.6 .763 
Third quartile 1,153.9 1,146.1 7.8 (-20.4, 35.9) 14.4 588 
Lowest quartile 1,116.8 1,112.7 4.1 (-21.9, 30.1) 13.2 .756 
Missing score 1,158.2 1,159.4 -1.2 (-29.8, 27.4) 14.6 .935 
F-test of difference” p=.513 
Host Institution Type 
Four-year college 1,171.4 1,168.5 3.0 (-22.3, 28.2) 12.9 .818 
Two-year college 1,171.4 1,147.0 24.4 (-15.6, 64.5) 20.4 .232 
Other 1,165.7 1,179.4 -13.7 (-73.4, 45.9) 30.4 .652 
F-test of difference” p=.533 


67 


Locale 


Rural 1,166.3 1,125.0 41.3 (28.0, 110.6) 35.3 .242 
City/suburb/town 1,172.5 1,169.2 3.3 (17.4, 24.0) 10.6 .756 
F-test of difference’ p=.302 


@ p-Values shown in this column are for tests of whether there was a statistically significant difference for the subgroup category in the row. 
» p-Values shown in this row are for a test of statistical differences between the categories of the subgroup in the rows above. 
Notes: Differences were calculated on the sample of students who enrolled in the first year. The outcome is the 75" percentile of SAT scores for incoming 
students at the college students attended. Find the Fit group mean and difference are estimated using the study’s regression model. The unadjusted standard 
deviations were 148.4 and 131.7 for the Find the Fit and regular UB advising groups, respectively. The SAT total score (math and verbal combined) can range from 
400-1600 with a mean of 1000 and a standard deviation of 200. To convert differences to an effect size the estimated difference can be divided by the standard 
deviation. For example, an estimated difference of 6.6 can be converted to an effect size of 0.03 (6.6/200). 
Sample Sizes: 

Overall difference: Find the Fit group = 986 students, Regular UB advising group = 825 students. 

Gender: Find the Fit group = 986 students, Regular UB advising group = 825 students. 

Race/ethnicity: Find the Fit group = 982 students, Regular UB advising group = 823 students. 

College entrance exam score quartiles: Find the Fit group = 986 students, Regular UB advising group = 825 students. 

Host institution type: Find the Fit group = 986 students, Regular UB advising group = 825 students. 

Locale: Find the Fit group = 986 students, Regular UB advising group = 825 students. 
Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


Likewise, there was no significant difference in the graduation rates of colleges attended by students in Find the 
Fit projects versus regular UB advising projects. Students in both Find the Fit projects and regular UB advising 
projects attended colleges with average graduation rates around 45 percent (Exhibit C.13). However, there were 
significant differences on this measure of college quality for two student subgroups: Hispanic students and 
students who were the most academically prepared. Among these subgroups of students, those in Find the Fit 
projects attended colleges with graduation rates that were about 4 to 7 percentage points higher, on average, 
than the colleges attended by their peers in regular UB advising projects. 


Exhibit C.13. Differences between Graduation Rates at the Colleges Attended by Find the Fit and 
Regular UB Advising Group Students, Overall and for Subgroups 


Overall Difference 


45.2 43.5 1.8 (-1.0, 4.5) 1.4 211 
Gender 
Female 46.2 43.6 2.6 (-0.3, 5.5) 1.5 .081 
Male 43.4 43.3 0.1 (3.2, 3.5) 1.7 .947 
F-test of difference” p=.085 
Race/Ethnicity 
Hispanic 46.3 42.0 4.3 (0.1, 8.2) 2.0 .037 
White, non-Hispanic 46.7 45.1 1.6 (-2.4, 6.0) DP 452 
Black, non-Hispanic 40.9 39.7 1.1 (-2.5, 4.8) 1.8 541 
Other, non-Hispanic 52.5 53.9 -1.4 (-6.6, 3.5) 2.6 592 
F-test of difference” p=.219 
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College Entrance Exam Score Quartile 

Highest quartile 68.3 61.4 6.9 (1.0, 12.8) 3.0 -022 
Second quartile 59.3 56.1 3.3 (-1.0, 7.7) DP 138 
Third quartile 48.9 47.5 1.4 (-2.4, 5.4) 2.0 479 
Lowest quartile 38.5 36.4 2.1 (-1.4, 5.3) 1.7 .232 
Missing score 38.3 38.6 -0.2 (-3.9, 3.6) 1.9 .903 
F-test of difference” p=.-2il 

Host Institution Type 

Four-year college 46.8 44.3 2.5 (-1.0, 6.1) 1.8 164 
Two-year college 39.7 39.8 -0.1 (-5.2, 5.1) 2.6 .970 
Other 48.6 46.3 2.3 (-6.1, 10.8) 4.3 585 
F-test of difference” p=.706 

Locale 

Rural 39.6 37.8 1.8 (6.9, 10.6) 4.5 .687 
City/suburb/town 45.7 44.0 1.7 (-1.2, 4.6) 1.5 .238 
F-test of difference” p=.990 


3 p-Values shown in this column are for tests of whether there was a statistically significant difference for the subgroup category in the row. 
> p-Values shown in this row are for a test of statistical differences between the categories of the subgroup in the rows above. 
Notes: Differences were calculated on sample of students who enrolled in the first year. The outcome is the graduation rate at the college students attended. Find 
the Fit group percentages and differences are estimated using the study’s regression model. The unadjusted standard deviations were 22.9 and 22.2 for the Find 
the Fit and Regular UB advising groups, respectively. 
Sample Sizes: 
Overall difference: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 
Gender: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 
Race/ethnicity: Find the Fit group = 1,418 students, Regular UB advising group = 1,330 students. 
College entrance exam score quartiles: Find the Fit group 1,422 students, Regular UB advising group = 1,333 students. 
Host institution type: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 
Locale: Find the Fit group = 1,422 students and Regular UB advising group = 1,333 students. 
Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


C.3 Persistence 


In the report, Exhibit 12 shows that Find the Fit had no significant effect on persistence into the third year. 
However, Find the Fit did have positive effects on persistence for three of the 16 exploratory subgroups 
examined-—students who were female, Hispanic, or among the less academically prepared (Exhibit C.14). These 
positive effect on persistence, measured as being continuously enrolled into the third fall after high school or 
completing a degree by that time, ranged from about 4 to 8 percentage points. Exhibit C.15 below shows that 
Find the Fit had no effect on persistence for the full sample regardless of the statistical model used for analysis. 
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Exhibit C.14 Effects on Continuous Enrollment into or Graduation by the Third Fall after High 
School, Overall and for Subgroups 


Overall Impact 

51.5 49.1 2.4 (-1.1, 5.9) 1.8 175 
Gender 
Female 55.1 50.9 4,2 (0.2, 8.2) 2.1 .042 
Male 45.6 46.1 -0.5 (-5.5, 4.5) 2.6 .831 
F-test of difference” p=.094 
Race/Ethnicity 
Hispanic 57.5 50.9 6.6 (0.2, 12.9) 3.2 .042 
White, non-Hispanic 49.5 48.8 0.7 (-5.8, 7.3) 3.4 .826 
Black, non-Hispanic 49.9 46.9 3.0 (-2.3, 8.3) 2.7 .273 
Other, non-Hispanic 50.1 53.1 -3.0 (-11.5, 5.6) 4.4 .497 
F-test of difference” p=.290 
College Entrance Exam Score Quartile 
Highest quartile 75.2 85.1 -9.9 (-22.2, 2.4) 6.3 114 
Second quartile 63.9 63.0 1.0 (-7.1, 9.0) Al 811 
Third quartile 56.4 57.6 -1.2 (-7.8, 5.5) 3.4 .734 
Lowest quartile 44.7 37.1 7.5 (2.6, 12.5) D5 .003 
Missing score 48.2 47.5 0.7 (-5.0, 6.4) 2.9 .806 
F-test of difference” p=.028 
Host Institution Type 
Four-year college 51.0 50.3 0.7 (-3.8, 5.3) 2.3 .759 
Two-year college 51.1 45.3 5.7 (-1.0, 12.5) 3.5 .097 
Other 53.7 50.4 3.3 (-7.0, 13.6) 5.3 528 
F-test of difference’ p= .474 
Locale 
Rural 43.4 46.0 -2.7 (-13.2, 7.8) 5.3 618 
City/suburb/town 52.5 49.5 3.1 (-0.7, 6.8) 1.9 106 
F-test of difference” p=.313 


* p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
> p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: The outcome is continuous enrollment in any college between July 1, 2016 (immediately after high school) and October 1, 2018, without an interruption in 
enrollment of five or more consecutive months, and/or graduation with any postsecondary degree or certificate by October 1, 2018. Find the Fit group 
percentages and impacts are estimated using the study’s regression model. 
Sample Sizes: 
Overall impact: Find the Fit t group = 2,336 students, Regular UB advising group = 2,107 students. 
Gender: Find the Fit group = 2,318 students, Regular UB advising group = 2,102 students. 
Race/ethnicity: Find the Fit group = 2,311 students, Regular UB advising group = 2,099 students. 
College entrance exam score quartiles: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 
Host institution type: Find the Fit t group = 2,336 students, Regular UB advising group = 2,107 students. 
Locale: Find the Fit t group = 2,336 students, Regular UB advising group = 2,107 students. 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 
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Exhibit C.15 Sensitivity Analyses for Effects on Continuous Enrollment into or Graduation by the 
Third Fall after High School 


Find the Fit Regular UB 
Group Advising Estimated Impact 
Students Group (95% Confidence Standard 
Model (%) Students (%) Interval) Error p-Value 

Linear regression (main 

model) 51.5 49.1 2.4 (-1.1, 5.9) 1.8 175 
Logistic regression 53.0 49.1 4.0 (-1.7, 9.6) 2.9 169 
Heteroscedasticity 

adjustment 51.5 49.1 2.4 (-1.3, 6.1) 1.9 428 
No covariates used in model 52.9 49.1 3.8 (-1.1, 8.8) 2.3 .220 


Notes: The outcome is continuous enrollment in any college between July 1, 2016 (immediately after high school) and October 1, 2018, without an interruption in 
enrollment of five or more consecutive months, and/or graduation with any postsecondary degree or certificate by October 1, 2018. Find the Fit group percentage 
and impact are estimated using the study’s regression model. 

Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


To explore whether Find the Fit affected postsecondary progress during the intervening years between the first 
and third falls after high school, the study conducted exploratory analyses to examine persistence at 
intermediate timepoints. Although there were no statistically significant impacts on progress through college, the 
difference in persistence between students in the Find the Fit and regular UB advising projects might have 
increased over time (Exhibit C.16). However, as Exhibits C.17 and C.18 below show, this difference was never 
statistically significant. That is, there were no statistically significant effects of Find the Fit on persistence 
throughout the first year (Exhibit C.17) or two years after high school (Exhibit C.18). The results were the same 
regardless of the statistical model used for analysis. 


Exhibit C.16 Impact of Find the Fit on Whether Students Enrolled and Persisted in College after 
High School 
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Notes: Data include 2,336 students in Find the Fit projects and 2,107 students in regular UB advising projects. Percentage of students represents the share of 
students who were (a) continuously enrolled in college from July 1, 2016 (immediately after high school graduation) through each time point (June 30, 2017 (Y1); 
June 30, 2018 (Y2); or October 1, 2018 (fall Y3)) without an interruption in enrollment of five or more consecutive months; or (b) graduated with any 
postsecondary degree or certificate by each time point June 30, 2017, or 2018, or October 1, 2018). Percentage for students in Find the Fit projects and impact are 
estimated using the study’s statistical model. UB is Upward Bound. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; NCES-Barron’s Admissions Competitiveness Index 2014. 
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Exhibit C.17 Overall Effect and Sensitivity Analyses for Effects on Continuous Enrollment through the 
First Year after High School 


Estimated 
Regular UB Impact 
Find the Fit Advising (95% 
Group Group Students Confidence Standard 
ee Model Students (%) (%) Interval) Error p-Value 
Linear regression (main model) 67.0 65.5 1.5 (-1.94.9) 1.7 386 
Logistic regression 68.9 65.5 3.3 (-5.2, 11.8) 4.3 Al 
Heteroscedasticity adjustment 67.0 65.5 1.5 (-2.2, 5.2) 1.9 428 
|No covariates used in model 68.4 65.5 2.8 (-1.7, 7.4) 2.3 .220 


Notes: The outcome is continuous enrollment in any college between July 1, 2016 (immediately after high school) and June 30, 2017, without an interruption in 
enrollment of five or more consecutive months, and/or graduation with any postsecondary degree or certificate by June 30, 2017. Find the Fit group percentage 
and impact are estimated using the study’s regression model. 

Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


Exhibit C.18 Overall Effect and Sensitivity Analyses for Effects on Continuous Enrollment throughout 
the First Two Years after High School 


Estimated | 
Regular UB Impact 
Find the Fit Advising (95% 
Group Group Students Confidence Standard 
; Model ___ Students (%) — (%) Interval) Error p-Value 
Linear regression (main model) 52.1 49.7 2.4 (-1.0, 5.9) 1.8 172 
Logistic regression 53.9 49.7 4.2 (-1.7, 10.1) 3.0 .167 
/Heteroscedasticity adjustment 52.1 49.7 2.4 (-1.2 6.0) 1.9 .193 
|No covariates used in model 53.6 49.7 3.8 (-1.1,8.7) 2.5 124 


Notes: The outcome is continuous enrollment in any college between July 1, 2016 (immediately after high school) and June 30, 2018, without an interruption in 
enrollment of five or more consecutive months, and/or graduation with any postsecondary degree or certificate by June 30, 2018. Find the Fit group percentage 
and impact are estimated using the study’s regression model. 

Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


C.4 Hypothesized Negative Consequences 

Though a shift to more selective colleges ideally would lead to improved later outcomes for students, it is also 
possible it could lead to unintended negative consequences for students.*° One potential negative consequence 
could be higher college costs as more selective colleges typically have a higher average posted cost (sometimes 
called “sticker price”) than less selective colleges do.*° However, more selective colleges often have more 
resources and are able to keep students’ costs down by making grants and scholarships available to low-income 
students such as those in Upward Bound. As a result, college selectivity does not necessarily correspond to 
higher out-of-pocket costs; that is, net prices.°’ Another possible negative consequence could occur if students 


Given the small increase in the number of students who enrolled in more selective colleges as a result of Find the Fit, any unintended negative 
consequences may not affect enough students to be detected. 

*6 On average, as college selectivity increases so too does the sticker price, based on author’s calculated correlation between the posted cost of attendance for 
the full academic year for first-time, full-time students in the IPEDS data and the NCES-Barron’s Admissions Competitiveness Index of the colleges. The 
correlation is 0.53 for in-state sticker price and 0.65 for out-of-state sticker price. 

College selectivity does not necessarily correspond to higher out-of-pocket costs, based on author’s calculated correlation between the estimated cost a 
student with a family income below $30,000 would pay to attend a college, after taking into account scholarships and grants, according to the IPEDS data 
and the NCES-Barron’s Admissions Competitiveness Index. The correlation is —-O.01. IPEDS provides the estimated net price for students with family 
incomes in different ranges. This study uses the lowest income range provided by IPEDS: under $30,000. Some 88 percent of students in the study are low 
income according to Upward Bound’s definition, which defines low income as being at or below 150 percent of the poverty line for the previous year. The 
poverty line varies by household size, but for households with three or more people, a family income below $30,000 qualified as low income at the time 
the study was conducted. 
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are less academically prepared than their peers, find themselves struggling academically, and so are more likely 
to drop out or transfer to a less selective college. 


C.4.1 Sticker Price 


In the report, Exhibit 10 shows the results from the exploratory analyses examining sticker price among students 
who enrolled in college. Students in Find the Fit projects attended colleges with higher average sticker prices 
than did students in regular UB advising projects. This was true for all of the subgroups examined, though not all 
of the differences were statistically significant (Exhibit C.19). The average difference in sticker price between the 
colleges attended by students in Find the Fit projects versus regular UB advising projects ranged from a high of 
more than $5,000 for the most academically prepared students to a low of about $500 for students who had no 
reported college entrance exam scores. 


Exhibit C.19 Differences between Sticker Prices at Colleges Attended by Find the Fit and Regular UB 
Advising Group Students, Overall and for Subgroups 


Overall Difference 


25,924.60 24,109.30 1,815.30 (324.8, 3305.8) 760.1 .O17 
Gender 
Female 26,245.50 24,364.80 1,880.70 (319.1, 3442.2) 796.3 .018 
Male 25,296.40 23,609.20 1,687.20 (-60.0, 3434.4) 891.0 .058 
F-test of difference” p=.783 
Race/Ethnicity 
Hispanic 25,128.20 23,900.80 1,227.40 (849.2, 3304.0) 1,059.0 .247 
White, non-Hispanic 26,249.30 23,904.30 2,345.00 (151.8, 4538.2) 1,118.4 .036 
Black, non-Hispanic 25,717.80 23,513.60 2,204.10 (295.4, 4112.8) 973.4 .024 
Other, non-Hispanic 27,225.00 26,511.50 713.50 (-1863.4, 3290.3) 1,314.1 587 
F-test of difference” p=.567 
College Entrance Exam Score Quartile 
Highest quartile 35,214.70 29,762.30 5,452.40 (2492.3, 8412.4) 1,509.5 000 
Second quartile 31,261.90 27,649.10 3,612.70 (1384.4, 5841.0) 1,136.4 .OO1 
Third quartile 26,723.10 25,055.40 1,667.70 (-352.4, 3687.8) 1,030.2 106 
Lowest quartile 24,056.80 22,518.90 1,537.90 (-237.1, 3312.8) 905.2 .089 
Missing score 22,762.10 22,214.60 547.50 (-1396.2, 2491.3) 991.2 581 
F-test of difference” p=.008 
Host Institution Type 
Four-year college 26,816.50 24,768.40 2,048.10 (97.1, 3999.0) 994.9 .040 
Two-year college 23,735.80 22,296.20 1,439.50 (-1346.8, 4225.9) 1,421.0 311 
Other 25,476.80 23,883.60 1,593.20 (-3027.5, 6213.9) 2,356.4 .499 
F-test of difference” p=.935 
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Locale 


Rural 24,063.60 20,803.10 3,260.50 (6213.8, 7992.8) 2,413.3 177 
City/suburb/town 26,060.70 24,404.20 1,656.50 (83.6, 3229.4) 802.1 .039 
F-test of difference” p=.528 


@ p-Values shown in this column are for tests of whether there was a statistically significant difference for the subgroup category in the row. 

» p-Values shown in this row are for a test of statistical differences between the categories of the subgroup in the rows above. 

Notes: Differences were calculated on the sample of students who enrolled in the first year. The outcome is the average posted cost of tuition, fees, and living 
expenses (“sticker price”) of colleges attended. Find the Fit group cost and difference are estimated using the study’s regression model. The unadjusted standard 
deviations were $10,614.1 and $9,924.9 for the Find the Fit and regular UB advising groups, respectively. 


Sample Sizes: 
Overall difference: Find the Fit group = 1,418 students, Regular UB advising group = 1,331 students. 
Gender: Find the Fit t group = 1,418 students, Regular UB advising group = 1,331 students. 
Race/ethnicity: Find the Fit group = 1,414 students, Regular UB advising group = 1,328 students. 
College entrance exam score quartiles: Find the Fit group = 1,418 students, Regular UB advising group = 1,331 students. 
Host institution type: Find the Fit group = 1,418 students, Regular UB advising group = 1,331 students. 
Locale: Find the Fit group = 1,418 students, Regular UB advising group = 1,331 students. 
Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


C.4.2 Net Price 


In the report, Exhibit 10 the results from the exploratory analyses examining net price. There was no significant 
difference in the likely out-of-pocket costs of college for students in Find the Fit projects versus regular UB 
advising projects. This was also true for most subgroups of students and projects examined (Exhibit C.20). The 
one exception was for students in the lowest quartile of college entrance exam scores. Among these students, 
those in Find the Fit projects attended colleges with a significantly higher average estimated net price than did 
those in regular UB advising projects (about $9,800 versus $8,800 per year). Therefore, Find the Fit could have 
led the least academically prepared students to attend colleges that were about $1,000 more expensive for them 
per year, on average, than the colleges they otherwise would have attended. 


Exhibit C.20 Differences between Net Price at Colleges Attended by Find the Fit and Regular UB 
Advising Group Students, Overall and for Subgroups 


Overall Difference 


10,183.30 9,586.90 596.40 (-156.9, 1349.6) 384.1 121 
Gender 
Female 10,389.50 9,688.10 701.40 (-92.8, 1495.6) 405.0 .083 
Male 9,779.90 9,389.00 390.90 (-508.1, 1289.9) 458.5 394 
F-test of difference” p=.4il 
Race/Ethnicity 
Hispanic 9,240.50 8,846.20 394.40 (-684.3, 1473.0) 550.1 .473 
White, non-Hispanic 10,320.20 9,753.90 566.30 (-573.2, 1705.7) 581.1 330 
Black, non-Hispanic 10,749.00 10,094.50 654.50 (-328.2, 1637.1) 501.1 192 
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691.5 181 


Other, non-Hispanic 10,023.00 9,098.50 924.50 (-431.4, 2280.4) 

F-test of difference” p=. 909 

College Entrance Exam Score Quartile 

Highest quartile 11,265.40 10,145.90 1,119.50 (-451.1, 2690.1) 801.0 162 
Second quartile 11,478.80 10,667.40 811.30 (-355.1, 1977.7) 594.8 .173 
Third quartile 10,430.50 10,299.30 131.20 (-917.4, 1179.7) 534.7 .806 
Lowest quartile 9,841.80 8,829.20 1,012.50 (101.7, 1923.4) 464.5 .029 
Missing score 9,510.40 9,338.60 171.80 (-833.3, 1176.9) 512.6 .738 
F-test of difference” p=.288 

Host Institution Type 

Four-year college 10,835.90 10,062.80 773.10 (-202.9, 1749.0) 497.7 120 
Two-year college 9,311.60 8,629.40 682.10 (-719.7, 2083.9) 714.9 .340 
Other 8,191.20 8,743.10 -552.00 (-2862.4, 1758.4) 1178.2 .639 
F-test of difference? p= .581 

Locale 

Rural 10,473.30 9,233.00 1,240.30 (-1159.4, 3639.9) 1,223.8 311 
City/suburb/town 10,144.30 9,618.40 525.90 (-268.7, 1320.6) 405.3 194 
F-test of difference” p=.579 


* p-Values shown in this column are for tests of whether there was a statistically significant difference for the subgroup category in the row. 
> p-Values shown in this row are for a test of statistical differences between the categories of the subgroup in the rows above. 
Notes: Differences were calculated on the sample of students who enrolled in the first year. The outcome is the average estimated cost of attendance after 
scholarships and grants (“net price”) for students from households with incomes below $30,000. Find the Fit group cost and difference are estimated using the 
study’s regression model. The unadjusted standard deviations were $5,313.1 and $5,093.4 for the Find the Fit and regular UB advising groups, respectively. 
Sample Sizes: 

Overall difference: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

Gender: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

Race/ethnicity: Find the Fit group = 1,418 students, Regular UB advising group = 1,330 students. 

College entrance exam score quartiles: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

Host institution type: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 

Locale: Find the Fit group = 1,422 students, Regular UB advising group = 1,333 students. 
Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 
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C.4.2 Dropout or Transfer to a Less Selective College 


In the report, Exhibit 11 shows the results from the exploratory analyses examining dropout or transfer to a less 
selective college. Students in Find the Fit projects were no more likely than those in regular UB advising projects 
to drop out of college or transfer to a less selective college by the second fall after high school. Likewise, there 
was no statistically significant impact of Find the Fit on dropout or transfer for any student or project subgroup 
(Exhibit C.21). 


Exhibit C.21_ Effects on Dropout or Transfer to a Less Selective College, Overall and for Subgroups 


Overall Impact 


18.7 20.0 -1.3 (-3.9, 1.4) 1.3 347 
Gender 
Female 17.9 20.5 -2.6 (-5.7, 0.6) 1.6 110 
Male 20.2 19.1 1.1 (-3.0, 5.2) 2.1 595 
F-test of difference” p=.136 
Race/Ethnicity 
Hispanic 15.6 20.3 -4.7 (-9.7, 0.4) 2.6 .069 
White, non-Hispanic 19.7 21.4 -1.7 (-6.9, 3.5) Doll 517 
Black, non-Hispanic 18.9 20.3 -1.4 (-5.5, 2.7) 2.1 .517 
Other, non-Hispanic 22.1 16.0 6.1 (-0.9, 13.1) 3.6 .088 
F-test of difference” p=.100 
College Entrance Exam Score Quartile 
Highest quartile 9.7 11.9 -2.2 (-12.8, 8.4) 5.4 .687 
Second quartile 17.3 20.6 -3.2 (-10.1,3.6) 3.5 354 
Third quartile 16.6 19.1 -2.5 (-8.0, 3.1) 2.8 384 
Lowest quartile 21.5 21.7 -0.3 (-4.3, 3.8) Dell .900 
Missing score 18.5 19.4 -0.9 (-5.6, 3.8) 2.4 713 
F-test of difference” p=.933 
Host Institution Type 
Four-year college 18.1 19.4 -1.2 (-4.6 2.1) 1.7 .470 
Two-year college 19.5 22.9 -3.4 (-8.6 1.9) Delt .208 
Other 20.2 17.3 2.9 (-4.7, 10.5) 3.9 453 
F-test of difference? p=.419 
Locale 
Rural 15.8 17.6 -1.7 (-9.8, 6.3) 41 .674 
City/suburb/town 19.1 20.3 -1.2 (-4.0, 1.6) 1.4 .399 
F-test of difference? p=.903 


4 p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 

> p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 

Notes: The outcome is the share of students who dropped out or transferred to a less selective school by the start of Year 2 (October 1, 2017). Find the Fit group 
percentage and impact are estimated using the study’s regression model. 
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Sample Sizes: 

Overall difference: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Gender: Find the Fit group = 2,318 students, Regular UB advising group = 2,102 students. 

Race/ethnicity: Find the Fit group = 2,311 students, Regular UB advising group = 2,099 students. 

College entrance exam score quartiles: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Host institution type: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Locale: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students for locale. 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; 
IPEDS 2015-16; baseline student survey 2015. 


Exhibit C.22 Sensitivity Analyses for Impact on Dropout or Transfer to a Less Selective College 


Regular UB Estimated 
Find the Fit Advising Impact 
Group Students Group (95% Confidence Standard 
Model (%) Students (%) Interval) Error  p-Value 
Linear regression (main model) 18.7 20.0 -1.3 (3.9, 1.4) 1.3 .347 
Logistic regression 18.4 20.0 -1.6 (-4.8, 1.7) 1.6 339 
Heteroscedasticity adjustment 18.7 20.0 -1.3 (-4.0, 1.5) 1.4 368 
No covariates used in model 18.4 20.0 -1.6 (-4.3, 1.2) 1.4 .259 


Notes: The outcome is the share of students who dropped out or transferred to a less selective college by the second fall after high school. Find the Fit group 
percentage and impact are estimated using the study’s regression model. 

Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; 
IPEDS 2015-16; baseline student survey 2015. 


C.5 Familiarity of the College Attended 


Because local and familiar colleges are often the default for low-income and first-generation college students, 
Find the Fit encouraged students to consider attending less-familiar colleges. The study examined whether there 
was an impact on student enrollment in a familiar college measured in two ways: (1) as the average distance of 
their college from the Upward Bound host institution and (2) as enrollment in the Upward Bound host 
institution. The results from these exploratory analyses are discussed below. 


C.5.1 Distance from Home 


There was no significant difference between students from Find the Fit projects versus regular UB advising 
projects in the average distance from the college they attended to their Upward Bound host institution (Exhibit 
C.23).°° However, Find the Fit might have encouraged Black, non-Hispanic students to attend colleges that were 
significantly farther from home: those in Find the Fit projects attended colleges that were an average of 127 miles 
from their host institution, compared to an average of 89 miles for Black, non-Hispanic students in regular UB 
advising projects. 


os Because the study did not have access to students’ home addresses, the address of students’ Upward Bound host institution (the institution where students 


participated in Upward Bound activities) was used instead. 
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Exhibit C.23 Differences in Distance from Home of Colleges Attended by Find the Fit and Regular UB 
Advising Group Students, Overall and for Subgroups 


Overall Difference 


116.4 102.1 14.2 (-10.1, 38.6) 12.4 252 
Gender 
Female 109.6 98.8 10.9 (-17.6, 39.3) 14.5 455 
Male 129.4 108.7 20.7 (-16.7, 58.8) 19.1 .278 
F-test of difference? p=.654 
Race/Ethnicity 
Hispanic 115.2 90.1 25.1 (-19.8,70.0) 22.9 .273 
White, non-Hispanic 105.6 116.2 -10.6 (-58.3, 37.0) 24.3 .662 
Black, non-Hispanic 127.4 88.6 38.8 (1.2, 76.4) 19.2 .043 
Other, non-Hispanic 99.3 138.6 -39.3 (-103.0, 24.4) 32.5 .226 
F-test of difference? p=.122 
College Entrance Exam Score Quartile 
Highest quartile 252.6 230.3 22.3 (-60.7, 105.2) 42.3 598 
Second quartile 135.1 104.7 30.5 (-26.2, 87.1) 28.9 292 
Third quartile 120.6 89.1 31.5 (-16.8, 79.8) 24.6 .200 
Lowest quartile 112.7 97.0 15.7 (-21.8, 53.1) 19.1 All 
Missing score 76.5 84.7 -8.2 (-52.2, 35.8) 22.4 715 
F-test of difference? p=.754 
Host Institution Type 
Four-year college 114.9 112.3 2.6 (-28.4, 33.5) 15.8 871 
Two-year college 107.7 70.9 36.8 (-12.2, 85.7) 25.0 141 
Other 133.6 105.5 28.1 (-44.5, 100.7) 37.0 .448 
F-test of difference? p=.470 
Locale 
Rural 144.0 131.3 12.8 (-71.5, 97.0) 43.0 .767 
City/suburb/town 113.9 99.6 14.3 (11.1, 39.8) 13.0 .270 
F-test of difference’ p=.972 


* p-Values shown in this column are for tests of whether there was a statistically significant difference for the subgroup category in the row. 
» p-Values shown in this row are for a test of statistical differences between the categories of the subgroup in the rows above. 
Notes: Differences were calculated on the sample of students who enrolled in the first year. Distance represents the average number of miles between the college 
attended and students’ Upward Bound host institution. Find the Fit group distance and difference are estimated using the study’s regression model. The 
unadjusted standard deviations were 274.0 and 278.6 for the Find the Fit and regular UB advising groups, respectively. 
Sample Sizes: 
Overall difference: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 
Gender: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 
Race/ethnicity: Find the Fit group = 1,419 students, Regular UB advising group = 1,333 students. 
College entrance exam score quartiles: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 
Host institution type: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 
Locale: Find the Fit group = 1,423 students, Regular UB advising group = 1,336 students. 
Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 
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C.5.2 Enrollment in Upward Bound Host Institution 


Exploratory analyses show that there was no significant difference between students from Find the Fit projects 
versus regular UB advising projects in attendance at their Upward Bound host institution after high school 
(Exhibit C.24 below). Find the Fit did have a statistically significant effect on this measure of institutional 
familiarity for three subgroups: female students; students who were not Black, White, or Hispanic; and students 
whose Upward Bound projects were hosted by two-year colleges. For example, among students whose Upward 
Bound projects were hosted by two-year colleges, those in Find the Fit projects were about 8 percentage points 
less likely to enroll in their host institution than their counterparts in regular UB advising projects. These results 
were not sensitive to the model specification (Exhibit C.25 below). 


Exhibit C.24 Effects on Enrollment at Upward Bound Host Institution, Overall and for Subgroups 


Overall Impact 

14.7 18.9 -4.3 (-8.6, 0.0) 2.2 .052 
Gender 
Female 14.7 19.4 -4.6 (-9.3, 0.0) 2.4 .049 
Male 14.7 18.1 -3.4 (-8.6, 1.7) 2.6 194 
F-test of difference” p=.579 
Race/Ethnicity 
Hispanic 13.1 17.0 -3.9 (-10.2, 2.5) 3.2 .232 
White, non-Hispanic 19.2 23.7 -4,.5 (-11.2, 2.1) 3.4 182 
Black, non-Hispanic 11.2 14.1 -2.8 (-8.6, 2.9) 2.9 .330 
Other, non-Hispanic 19.6 27.6 -8.1 (-15.9, -O.2) 4.0 .045 
F-test of difference” p=.681 
College Entrance Exam Score Quartile 
Highest quartile 17.3 26.7 -9.4 (-19.7, 0.9) 5.2 .073 
Second quartile 12.0 17.3 -5.3 (-12.4 1.9) 3.7 .151 
Third quartile 11.5 17.2 -5.8 (-12.0, 0.4) 3.2 .068 
Lowest quartile 14.4 16.1 -1.6 (-6.8 ,3.6) 2.6 539 
Missing score 17.6 23.0 -5.4 (11.1, 0.2) 2.9 .O61 
F-test of difference’ p=.414 
Host Institution Type 
Four-year college 13.7 17.6 -3.9 (-8.8, 1.0) 2.5 122 
Two-year college 24.4 32.0 -7.6 (-14.6, -0.6) 3.6 .034 
Other 0.3 0.0 0.3 (-10.9, 11.5) 5.7 .958 
F-test of difference” p=.480 
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Regular UB | 
Find the Fit Advising 
Group Group Estimated Impact 
Students Students (95% Confidence Standard 

(%) (%) Interval) _ Error p-Value* | 
Locale 
Rural 20.1 30.1 -10.0 (-22.7, 2.7) 6.5 122 
City/suburb/town 14.0 17.5 -3.5 (-8.1, 1.1) 2.3 133 
F-test of difference? p=.347 


a p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
> p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: The outcome is the share of students who enrolled in their Upward Bound host institution. Find the Fit group percentage and impact are estimated using 
the study’s regression model. 
Sample Sizes: 

Overall impact: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Gender: Find the Fit group = 2,318 students, Regular UB advising group = 2,102 students. 

Race/ethnicity: Find the Fit group = 2,311 students, Regular UB advising group = 2,099 students. 

College entrance exam score quartiles: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students 

Host institution type: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Locale: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 
Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


Exhibit C.25 Sensitivity Analyses for Impact on Familiarity of the College Attended 


Find the Fit Regular UB 
Group Advising Estimated Impact 
Students Group Students (95% Confidence Standard 
| Model (%) (%) Interval) Error p-Value_ 
Linear regression (main model) 14.7 18.9 -4.3 (-8.6, 0.0) 2.2 .052 
Logistic regression 14.5 18.9 -4.5 (-8.7, -0.2) 2.2 .041 
Heteroscedasticity adjustment 14.7 18.9 -4.3 (-8.8, 0.3) 2.3 .064 
No covariates used in model 14.6 18.9 -4.3 (8.7, 0.1) DP .054 


Notes: The outcome is the share of students who enrolled in their Upward Bound host institution. Find the Fit group percentage and impact are estimated using 
the study’s regression model. 

Sample Size: Find the Fit group = 2,336 students, Regular UB advising group = 2,107 students. 

Source: FSA 2017; NSC 2016; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


C.6 Relational Analyses 


Some research suggests that enrolling in a more selective college is associated with higher rates of college 
persistence. °° This relationship is likely due in part to differences in the characteristics of students who attend 
different types of schools, including the stronger academic credentials of students at more selective colleges, but could 
also be due to the institutional resources more selective colleges can provide to support students in persisting. °° This 
study explored, through descriptive analyses, some of the hypothesized benefits of enrolling in a more selective 
college and of college match, looking specifically at how initial college choices are related to longer-term persistence. 


Exhibits C.26 and C.27 below show that after adjusting for student characteristics such as pre-college academic 
achievement, students who initially enrolled in more selective colleges were generally more likely to be enrolled 
continuously into their third year than were students who initially enrolled in less selective colleges. Differences 
between students enrolled in colleges with adjacent selectivity levels were often not statistically significant. 
However, there were differences of about 8 to 12 percentage points, in persistence rates for two-year versus four- 


Some, but not all of the research examining the relationship between college selectivity and persistence or completion accounts for students’ pre-college 
characteristics, such as GPA, test scores, and advanced course-taking, that are also likely related to students’ success in college. 

The difference in instructional expenditures between most competitive colleges and all other selectivity levels is stark: most competitive colleges spend an 
average of $34,000 per student, whereas highly competitive colleges spend an average of $16,000 and very competitive colleges an average of $11,000. Two- 
year colleges as well as somewhat competitive and competitive four-year colleges only spend about $7,000 to $9,000 per student (author’s calculations using 
data from the NCES-Barron’s Admissions Competitiveness Index and IPEDS). 
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year colleges, and for other four-year colleges (that is, noncompetitive or unranked four-year colleges) versus 
somewhat competitive colleges (Exhibit C.27 below). For example, there was no statistically significant difference 
in continuous enrollment for students enrolled in most competitive versus highly competitive colleges. However, 
there was a significant difference in continuous enrollment for students who attended most competitive colleges 
versus competitive colleges. There was also a significant difference for students enrolled in a highly competitive 
college, a very competitive college, or a competitive college versus those enrolled in an any four-year college, 
which includes noncompetitive, specialty, and unranked colleges. 


Exhibit C.26 Persistence into the Third Fall after High School, by Selectivity of the College Attended 


in the First Fall 

Most competitive 85 85.5 

Highly competitive 147 80.0 73.6 86.4 
Very competitive 446 77.6 73.7 81.5 
Competitive 1,072 73.2 70.6 75.7 
Somewhat competitive 324 73.9 69.6 78.3 
Four-year college 295 62.1 57.6 66.6 
Two-year college 909 54.6 51.9 57.3 
Not enrolled 1,165 6.8 4.3 9.4 
F-test of difference’ p=<.001 


CL is confidence limit for the 95 percent confidence interval. A confidence interval is a measure of the precision of the impact estimate. If an analysis was repeated many 
times with different students, the estimated impact would be expected to fall within the confidence interval 95 percent of the time. 

The “adjusted mean” refers to the model-adjusted mean percentage of students estimated from a regression model that includes the same set of student and project 
characteristics used in the impact analysis models (except for treatment status). 

bp-Values shown in this row are for a test of whether adjusted means statistically differed across the selectivity levels in the rows above. 

Notes: The outcome is continuous enrollment in any college between July 1, 2016, and October 1, 2018, without an interruption in enrollment of five or more consecutive 
months, and/or graduation with any postsecondary degree or certificate by October 1, 2018. 

Sample Size: 4,443 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 
2015-16; baseline student survey 2015. 


Exhibit C.27 Differences in Persistence into the Third Fall after High School, by Selectivity of the 
College Attended in the First Fall 


Most competitive compared to 


Highly competitive 5.5 (-4.5, 15.6) 5.1 .279 
Very competitive 7.9 (-0.9, 16.7) 4.5 .079 
Competitive 12.3 (3.8, 20.9) 4.4 .005 
Somewhat competitive 11.6 (2.3, 20.9) 4.7 .O15 
Other four-year college 23.4 (14.0, 32.8) 48 <.0001 
Two-year college 30.9 (22.0, 39.7) 4.5 <.0001 
Not enrolled 78.7 (69.8, 87.5) 4.5 <.0001 
Very competitive 2.4 (-4.7, 9.4) 3.6 509 
Competitive 6.8 (0.2, 13.4) 3.4 .043 
Somewhat competitive 6.0 (-1.5, 13.6) 3.8 116 
Other four-year college 17.9 (10.2, 25.5) 3.9 <.0001 
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3.5 <.0001 


Two-year college 25.4 (18.5, 32.2) 

Not enrolled 73.1 (66.3, 80.0) 3.5 <,0001 
Competitive 4.4 (0.2, 8.7) 2.2 .041 
Somewhat competitive 3.7 1.9, 9.2) 2.8 197 
Other four-year college 15.5 (9.8, 21.2) 2.9 <.0001 
Two-year college 23.0 (18.4, 27.6) 2.4 <.0001 
Not enrolled 70.7 (66.2, 75.2) 2.3 <.0001 
Somewhat competitive -0.8 (-5.5, 4.0) 2.4 .749 
Other four-year college 11.1 (6.1, 16.0) 2.5 <.0001 
Two-year college 18.5 (15.1, 22.0) 1.8 <.0001 
Not enrolled 66.3 (63.0, 69.6) 1.7 <.0001 
Other four-year college 11.8 (5.8, 17.9) 3.1 .000 
Two-year college 19.3 (14.4, 24.2) 2.5 <.0001 
Not enrolled 67.1 (62.3, 71.9) 2.4 <.0001 
Other four-year college compared to 

Two-year college 7.5 (2.4, 12.6) 2.6 .004 
Not enrolled 55.3 (50.3, 60.2) 2.5 <.0001 
Two-year college compared to 

Not enrolled 47.8 (44.4, 51.1) 1.7 <.0001 


@ This column shows the difference in the percentage of students who are continuously enrolled in colleges at the specified selectivity levels. For example, the percentage of 
students continuously enrolled into the third year at most competitive colleges is 5.5 percentage points higher than at highly competitive colleges. 

Notes: The outcome is continuous enrollment in any college between July 1, 2016, and October 1, 2018, without an interruption in enrollment of five or more consecutive 
months, and/or graduation with any postsecondary degree or certificate by October 1, 2018. 

Sample Size: 4,443 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; NCES-Barron’s Admissions Competitiveness Index 2014; IPEDS 2015-16; APR 2014-15; college entrance exam score data 


2015; baseline student survey 2015. 


The study also examined whether being undermatched or overmatched is related to students’ likelihood of 
persisting in college. Exhibit C.28 below suggests that, for students at all levels of academic achievement— 
highest, moderate, and lowest®'—the likelihood of persisting in college was highest when they attended the most 
selective college possible, regardless of their match status. This was particularly true for lower-achieving 
students. Among these students, about two-thirds of those who overmatched were continuously enrolled into 
the third fall, compared to fewer than half of students with similar achievement who matched. For students with 
moderate academic achievement, their likelihood of persistence was about the same regardless of whether they 
were overmatched or matched. However, for these students, undermatching carried a particularly steep penalty: 
slightly more than half of students with moderate achievement levels who undermatched persisted into the third 
fall, compared to more than three-quarters of students in this group who attended a matched college. The 
highest-achieving students had a very high chance of persisting into the third fall regardless of the college they 
initially attended. 


| Three academic achievement groups were created based on students’ cumulative GPA, AP/IB course taking, and college entrance exam scores. 
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Exhibit C.28 Persistence into the Third Fall after High School, by Student Achievement and 
Undermatch Status 


Highest-Achieving Students 


Match 262 89.3 84.4 94.2 
Undermatch 232 85.8 80.7 90.9 
Overmatch 232 81.5 76.4 86.6 
Match 487 79.5 75.9 83.1 
Undermatch 177 57.4 51.6 63.1 
Overmatch 1202 69.2 66.7 71.7 
Match 686 48.5 45.5 51.6 
F-test of difference’ p=<.001 


CL is confidence limit for the 95 percent confidence interval. A confidence interval is a measure of the precision of the impact estimate. If an analysis was 
repeated many times with different students, the estimated impact would be expected to fall within the confidence interval 95 percent of the time. 

@The “adjusted mean” refers to the model-adjusted mean percentage of students estimated from a regression model that includes the same set of student and 
project characteristics used in the impact analysis models (except for treatment status). 

bp-Values shown in this row are for a test of whether adjusted means statistically differed across the categories in all of the rows above. 

Notes: The outcome is continuous enrollment in any college between July 1, 2016, and October 1, 2018, without an interruption in enrollment of five or more 
consecutive months, and/or graduation with any postsecondary degree or certificate by October 1, 2018. 

Sample Size: 3,278 students who enrolled in college in the first fall after high school. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; 
IPEDS 2015-16; baseline student survey 2015. 


Finally, the study examined whether enrolling in a familiar college—that is, students’ Upward Bound host 
institution—was associated with persistence. Historically, a substantial share of students in Upward Bound end 
up attending the college that hosts their Upward Bound project.®” However, these host colleges might or might 
not be a good academic fit for individual students. Although a lower percentage of students who enrolled in their 
host institution persisted into the third fall after high school than their peers who enrolled in less familiar 
colleges—that is, colleges other than their host institution—the difference was small in magnitude and not 
statistically significant (Exhibit C.29). 


Exhibit C.29 Differences in Continuous Enrollment into the Third Fall after High School, by Whether 
Students Attended a Familiar College 


Continuous enrollment into 
the third fall after high school 


Notes: The outcome is continuous enrollment in any college between July 1, 2016, and October 1, 2018, without an interruption in enrollment of five or more 
consecutive months, and/or graduation with any postsecondary degree or certificate by October1, 2018. 

Sample Sizes: Attended a familiar college (their Upward Bound host institution) = 742 students, Attended an unfamiliar college (not their Upward Bound host 
institution) = 2,536 students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015. 


67.4 71.0 3.6 (-0.2, 5.5) 1.9 .062 


- Martinez et al. 2018. 
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SECTION D. EFFECTS BY LEVEL OF FIND THE FIT IMPLEMENTATION 


This section presents results from the exploratory analyses that examined whether the effect of Find the Fit 
varied by how much projects used Find the Fit. There were no significant differences in the effect of Find the Fit 
by level of implementation for any of the main outcome measures: undermatch (Exhibit D.1), selectivity level of 
the college attended (Exhibit D.2), or persistence into the third fall (Exhibit D.3 below). The lack of consistent 
differences across implementation levels could suggest that the implementation measures the study created 
mask important differences in how projects implemented Find the Fit or they could derive from the flexibility 
that Upward Bound projects had to use Find the Fit as they deemed best. © 


Exhibit D.1| Effects on Undermatch, by Level of Find the Fit Implementation 


Low implementers 42.3 35.2 7.1(-5.9, 20.1) 6.6 .284 
Moderate implementers 37.7 39.9 -2.3 (-8.9, 4.37) 3.4 504 
High implementers 40.1 43.2 -3.1¢-10.1, 3.9) 3.6 388 
F-test of difference’ p=.385 


’p-Values shown in this column are for tests of whether there was a statistically significant impact for the implementation category in the row. 
bp-Values shown in this row are for a test of whether impacts statistically differed between the implementation categories in the rows above. 
Notes: Percentage represents the share of students who undermatched in their college choice. Find the Fit group percentage and impact are estimated using the study’s 
regression model. Find the Fit group projects were categorized as low implementers if they implemented less than 25 percent of any one Find the Fit component; moderate 
implementers if they implemented more than a quarter but not necessarily 75 percent of each component; and high implementers if they implemented 75 percent of more 
of each component. 
Sample Sizes: 

Low implementers: Find the Fit group = 262 students, Regular UB advising group = 230 students. 

Moderate implementers: Find the Fit group = 1,235 students, Regular UB advising group = 859 students. 

High implementers: Find the Fit group = 777 students, Regular UB advising group = 907 students. 
Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student 
survey 2015; project survey 2016; program monitoring data 2015-16. 


Exhibit D.2.—_ Effects on Selectivity Level of the College Attended, by Level of Find the Fit 
Implementation 


Most Competitive 


Low implementers 3.2 2.1 1.1 2.1, 4.2) 1.6 .497 
Moderate implementers 2.0 1.4 0.6 (-1.0, 2.2) 0.8 454 
High implementers 2.2 1.0 1.1 0.6, 2.8) 0.9 183 
F-test of difference’ p=.894 

Low implementers 6.7 4.7 2.0 (-4.2, 8.2) 3.2 518 
Moderate implementers 7.2 3.8 3.4 (0.2, 6.5) 1.6 .036 
High implementers 6.5 3.5 3.0 (-0.3, 6.3) 1.7 .077 
F-test of difference? p=.934 


63 Limited statistical power could also explain the lack of statistically significant findings. 
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At Least Very Competitive 


Low implementers 14.5 14.9 -0.4 (9.5, 8.7) 4.6 .937 

Moderate implementers 15.5 14.4 1.1 (-3.5, 5.7) 2.3 .642 

High implementers 17.7 11.4 6.3 (1.4, 11.2) DS) .012 

F-test of difference’ p=.237 

Low implementers 32.6 33.6 -1.0 (-14.0, 12.0) 6.6 .880 
Moderate implementers 40.6 38.6 2.0 (-4.6, 8.6) 3.4 555 

High implementers 38.0 38.1 0.0 (-7.1, 7.0) 3.6 991 

F-test of difference’ p=.884 

Low implementers 35.1 41.7 -6.6 (-19.8, 6.7) 6.8 331 

Moderate implementers 49.2 44.5 4.7 (-2.0, 11.5) 3.4 168 

High implementers 46.3 441 2.2 (-4.9, 9.4) 3.6 542 
F-test of difference? p=.335 

Low implementers 38.2 44.3 -6.0 (-18.2, 6.1) 6.2 331 
Moderate implementers 51.9 53.2 -1.3 7.4, 4.9) 3.1 .685 
High implementers 50.6 50.6 -0.0 (-6.6, 6.4) 3.3 .996 
F-test of difference’ p=.697 

Low implementers 68.9 74.9 -6.0 (-16.8, 4.7) 55 .269 
Moderate implementers 70.7 73.6 -2.8 (-8.2, 2.6) 2.7 306 
High implementers 70.7 69.7 1.0 (-4.7, 6.7) 2.9 .737 
F-test of difference’ p=.445 


p-Values shown in this column are for tests of whether there was a statistically significant impact for the implementation category in the row. 
>p-Values shown in this row are for a test of whether impacts statistically differed between the implementation categories in the rows above. 
Notes: Percentage represents the share of students who attended a college of at least a given selectivity level. For example, “at least very competitive” includes attending 
colleges at the two selectivity levels above that: highly competitive and most competitive. Differences were compared at each level by combining students who had attended 
colleges at that selectivity level and the levels above. Find the Fit group percentage and impact are estimated using the study’s regression model. Find the Fit group projects 
were categorized as low implementers if they implemented less than 25 percent of any one Find the Fit component; moderate implementers if they implemented more than 
a quarter but not necessarily 75 percent of each component; and high implementers if they implemented 75 percent of more of each component. 
Sample Sizes: 

Low implementers: Find the Fit group = 267 students, Regular UB advising group = 235 students. 

Moderate implementers: Find the Fit group = 1,267 students, Regular UB advising group = 919 students. 

High implementers: Find the Fit group = 802 students, Regular UB advising group = 953 students. 
Source: FSA 2017; NSC 2016; NCES-Barron’s Admissions Competitiveness Index 2014; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student 
survey 2015; project survey 2016; program monitoring data 2015-16. 


Exhibit D.3. Effects of Find the Fit on Persistence into the Third Fall after High School, by Level of 
Find the Fit Implementation 


Low implementers 50.1 51.5 -1.4 (-11.9, 9.1) 5.4 .794 
Moderate implementers 52.5 50.7 1.8 (3.4, 7.1) 2.7 501 
High implementers 50.9 46.9 4.0 (-1.6, 9.6) 2.9 165 
F-test of difference? p=.657 


4p-Values shown in this column are for tests of whether there was a statistically significant impact for the implementation category in the row. 
p-Values shown in this row are for a test of whether impacts statistically differed between the implementation categories in the rows above. 
Notes: The outcome is continuous enrollment in any college between July 1, 2016, and October 1, 2018, without an interruption in enrollment of five or more consecutive 
months, and/or graduation with any postsecondary degree or certificate by October 1, 2018. Find the Fit group percentage and impact are estimated using the study’s 
regression model. Find the Fit group projects were categorized as low implementers if they implemented less than 25 percent of any one Find the Fit component; moderate 
implementers if they implemented more than a quarter but not necessarily 75 percent of each component; and high implementers if they implemented 75 percent of more 
of each component. 
Sample Sizes: 

Low implementers: Find the Fit group = 267 students, Regular UB advising group = 235 group students. 

Moderate implementers: Find the Fit group = 1,267 students, Regular UB advising group = 919 students. 

High implementers: Find the Fit group = 802 students, Regular UB advising group = 953 students. 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; APR 2014-15; college entrance exam score data 2015; IPEDS 2015-16; baseline student survey 2015; project survey 2016; 
monitoring data 2015-16. 
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